2024-03-29T10:49:37.883 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T10:49:37.883 DEBUG:teuthology.run:Teuthology command: teuthology --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622670 --owner scheduled_amathuri@teuthology --verbose --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622670/orig.config.yaml 2024-03-29T10:49:37.918 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:49:38.002 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622670 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622670' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi071.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOcs4iTZhsRuZeUJNRGglRkljVWYqSRuS0hdE/+KYKineMcL/m/NgmcjEif3aIR8bvEZPniQYgZqCuqPgWkrFpw= smithi137.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLfH/EJ2PkJNYZoxy9WM5+7uIdEXJwM+CpijWnhtSm97Q9XamauYkrsr/bGxV6otwT2Zv1yDdYVkkOOT1KktJe0= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T10:49:38.003 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T10:49:38.003 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T10:49:38.003 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T10:49:38.005 INFO:teuthology.task.internal:Checking packages... 2024-03-29T10:49:38.028 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T10:49:38.028 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T10:49:38.028 INFO:teuthology.packaging:ref: None 2024-03-29T10:49:38.028 INFO:teuthology.packaging:tag: None 2024-03-29T10:49:38.028 INFO:teuthology.packaging:branch: main 2024-03-29T10:49:38.029 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:49:38.029 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T10:49:38.414 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T10:49:38.415 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T10:49:38.421 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T10:49:38.421 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T10:49:38.452 INFO:teuthology.task.internal:Saving configuration 2024-03-29T10:49:38.479 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T10:49:38.485 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:49:38.510 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622670', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:44:35.638200', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOcs4iTZhsRuZeUJNRGglRkljVWYqSRuS0hdE/+KYKineMcL/m/NgmcjEif3aIR8bvEZPniQYgZqCuqPgWkrFpw='} 2024-03-29T10:49:38.532 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622670', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:44:35.640092', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLfH/EJ2PkJNYZoxy9WM5+7uIdEXJwM+CpijWnhtSm97Q9XamauYkrsr/bGxV6otwT2Zv1yDdYVkkOOT1KktJe0='} 2024-03-29T10:49:38.532 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T10:49:38.539 INFO:teuthology.task.internal:roles: ubuntu@smithi071.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T10:49:38.539 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T10:49:38.539 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T10:49:38.634 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5e885c00d0>, signals=[15]) 2024-03-29T10:49:38.635 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T10:49:38.641 INFO:teuthology.task.internal:Opening connections... 2024-03-29T10:49:38.641 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi071.front.sepia.ceph.com 2024-03-29T10:49:38.642 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:49:38.734 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi137.front.sepia.ceph.com 2024-03-29T10:49:38.735 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:49:38.829 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T10:49:38.835 DEBUG:teuthology.orchestra.run.smithi071:> uname -m 2024-03-29T10:49:38.854 INFO:teuthology.orchestra.run.smithi071.stdout:x86_64 2024-03-29T10:49:38.855 DEBUG:teuthology.orchestra.run.smithi071:> cat /etc/os-release 2024-03-29T10:49:38.911 INFO:teuthology.orchestra.run.smithi071.stdout:NAME="CentOS Stream" 2024-03-29T10:49:38.911 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION="9" 2024-03-29T10:49:38.911 INFO:teuthology.orchestra.run.smithi071.stdout:ID="centos" 2024-03-29T10:49:38.911 INFO:teuthology.orchestra.run.smithi071.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:49:38.911 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_ID="9" 2024-03-29T10:49:38.911 INFO:teuthology.orchestra.run.smithi071.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:49:38.911 INFO:teuthology.orchestra.run.smithi071.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:49:38.912 INFO:teuthology.orchestra.run.smithi071.stdout:ANSI_COLOR="0;31" 2024-03-29T10:49:38.912 INFO:teuthology.orchestra.run.smithi071.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:49:38.912 INFO:teuthology.orchestra.run.smithi071.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:49:38.912 INFO:teuthology.orchestra.run.smithi071.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:49:38.912 INFO:teuthology.orchestra.run.smithi071.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:49:38.912 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:49:38.912 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:49:38.912 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2024-03-29T10:49:38.930 DEBUG:teuthology.orchestra.run.smithi137:> uname -m 2024-03-29T10:49:38.948 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2024-03-29T10:49:38.948 DEBUG:teuthology.orchestra.run.smithi137:> cat /etc/os-release 2024-03-29T10:49:39.004 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="CentOS Stream" 2024-03-29T10:49:39.004 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="9" 2024-03-29T10:49:39.004 INFO:teuthology.orchestra.run.smithi137.stdout:ID="centos" 2024-03-29T10:49:39.004 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:49:39.004 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="9" 2024-03-29T10:49:39.004 INFO:teuthology.orchestra.run.smithi137.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:49:39.004 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:49:39.004 INFO:teuthology.orchestra.run.smithi137.stdout:ANSI_COLOR="0;31" 2024-03-29T10:49:39.004 INFO:teuthology.orchestra.run.smithi137.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:49:39.004 INFO:teuthology.orchestra.run.smithi137.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:49:39.004 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:49:39.004 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:49:39.005 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:49:39.005 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:49:39.005 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2024-03-29T10:49:39.022 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T10:49:39.030 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T10:49:39.058 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T10:49:39.058 DEBUG:teuthology.orchestra.run.smithi071:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:49:39.061 DEBUG:teuthology.orchestra.run.smithi137:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:49:39.076 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T10:49:39.083 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T10:49:39.083 DEBUG:teuthology.orchestra.run.smithi071:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:49:39.117 DEBUG:teuthology.orchestra.run.smithi137:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:49:39.167 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T10:49:39.219 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T10:49:39.234 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T10:49:39.234 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T10:49:39.234 DEBUG:teuthology.orchestra.run.smithi071:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:49:39.235 DEBUG:teuthology.orchestra.run.smithi137:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:49:39.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:49:39.251 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-03-29T10:49:39.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:49:39.252 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2024-03-29T10:49:39.306 INFO:teuthology.orchestra.run.smithi137.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:49:39.306 INFO:teuthology.task.kernel:Running kernel on smithi137: 5.14.0-361.el9.x86_64 2024-03-29T10:49:39.307 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum install -y kernel 2024-03-29T10:49:39.309 INFO:teuthology.orchestra.run.smithi071.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:49:39.309 INFO:teuthology.task.kernel:Running kernel on smithi071: 5.14.0-361.el9.x86_64 2024-03-29T10:49:39.309 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum install -y kernel 2024-03-29T10:49:41.832 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - AppStream 7.1 MB/s | 6.7 MB 00:00 2024-03-29T10:49:41.958 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS-9 - AppStream 6.4 MB/s | 6.7 MB 00:01 2024-03-29T10:49:43.224 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-29T10:49:43.608 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-29T10:49:44.065 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-29T10:49:47.166 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - AppStream 9.3 MB/s | 19 MB 00:02 2024-03-29T10:49:47.532 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - BaseOS 2.2 MB/s | 8.0 MB 00:03 2024-03-29T10:49:58.432 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - CRB 810 kB/s | 6.1 MB 00:07 2024-03-29T10:49:59.957 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 16 kB 00:00 2024-03-29T10:50:03.925 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 5.4 MB/s | 21 MB 00:03 2024-03-29T10:50:08.966 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-03-29T10:50:09.754 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - AppStream 914 kB/s | 19 MB 00:21 2024-03-29T10:50:11.248 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:50:11.248 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:50:11.343 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T10:50:11.349 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:50:11.349 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-29T10:50:11.349 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:50:11.349 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T10:50:11.349 INFO:teuthology.orchestra.run.smithi071.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:50:11.349 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-29T10:50:11.350 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:50:11.350 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:50:11.350 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:50:11.350 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:50:11.350 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T10:50:11.350 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:50:11.350 INFO:teuthology.orchestra.run.smithi071.stdout:Install 4 Packages 2024-03-29T10:50:11.350 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:50:11.351 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 76 M 2024-03-29T10:50:11.351 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 123 M 2024-03-29T10:50:11.351 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T10:50:11.584 INFO:teuthology.orchestra.run.smithi071.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 822 kB/s | 192 kB 00:00 2024-03-29T10:50:12.276 INFO:teuthology.orchestra.run.smithi071.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 16 MB/s | 15 MB 00:00 2024-03-29T10:50:13.151 INFO:teuthology.orchestra.run.smithi071.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 19 MB/s | 34 MB 00:01 2024-03-29T10:50:13.797 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-29T10:50:14.234 INFO:teuthology.orchestra.run.smithi071.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 10 MB/s | 28 MB 00:02 2024-03-29T10:50:14.237 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:14.238 INFO:teuthology.orchestra.run.smithi071.stdout:Total 26 MB/s | 76 MB 00:02 2024-03-29T10:50:14.238 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T10:50:14.595 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T10:50:14.595 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T10:50:15.231 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-03-29T10:50:15.417 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T10:50:15.417 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T10:50:16.299 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-29T10:50:17.711 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T10:50:18.334 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:50:18.429 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:50:21.232 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-03-29T10:50:21.267 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:50:21.410 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:50:23.472 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:50:23.473 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:50:23.565 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T10:50:23.577 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:50:23.577 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-29T10:50:23.578 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:50:23.578 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T10:50:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:50:23.578 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-29T10:50:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:50:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:50:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:50:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:50:23.578 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T10:50:23.578 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:50:23.578 INFO:teuthology.orchestra.run.smithi137.stdout:Install 4 Packages 2024-03-29T10:50:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:50:23.578 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 76 M 2024-03-29T10:50:23.579 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 123 M 2024-03-29T10:50:23.579 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T10:50:23.717 INFO:teuthology.orchestra.run.smithi137.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-03-29T10:50:24.309 INFO:teuthology.orchestra.run.smithi137.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 21 MB/s | 15 MB 00:00 2024-03-29T10:50:24.984 INFO:teuthology.orchestra.run.smithi137.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 24 MB/s | 34 MB 00:01 2024-03-29T10:50:25.317 INFO:teuthology.orchestra.run.smithi137.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 17 MB/s | 28 MB 00:01 2024-03-29T10:50:25.320 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:25.321 INFO:teuthology.orchestra.run.smithi137.stdout:Total 44 MB/s | 76 MB 00:01 2024-03-29T10:50:25.321 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T10:50:25.709 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T10:50:25.709 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T10:50:26.101 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:50:26.142 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:50:26.583 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T10:50:26.583 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T10:50:28.885 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T10:50:29.514 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:50:29.601 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:50:30.911 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:50:32.357 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:50:32.494 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:50:37.233 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:50:37.351 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:50:42.132 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:51:00.523 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:51:00.523 INFO:teuthology.orchestra.run.smithi071.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:51:00.523 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:51:00.565 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:51:00.899 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:51:00.899 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:51:00.899 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:51:00.900 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:51:01.312 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:51:01.312 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:51:01.313 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T10:51:01.313 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:51:01.313 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:51:01.313 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:51:01.313 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:51:01.313 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:51:01.313 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T10:51:01.654 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:51:01.755 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:51:01.756 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:51:01.756 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:51:01.756 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:51:01.756 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:51:01.756 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T10:51:01.756 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi071.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:51:01.756 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:51:01.837 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:51:01.837 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:51:01.837 DEBUG:teuthology.orchestra.run.smithi071:> sudo rpm -qi grub2-tools 2024-03-29T10:51:01.871 INFO:teuthology.orchestra.run.smithi071.stdout:Name : grub2-tools 2024-03-29T10:51:01.871 INFO:teuthology.orchestra.run.smithi071.stdout:Epoch : 1 2024-03-29T10:51:01.871 INFO:teuthology.orchestra.run.smithi071.stdout:Version : 2.06 2024-03-29T10:51:01.871 INFO:teuthology.orchestra.run.smithi071.stdout:Release : 68.el9 2024-03-29T10:51:01.871 INFO:teuthology.orchestra.run.smithi071.stdout:Architecture: x86_64 2024-03-29T10:51:01.871 INFO:teuthology.orchestra.run.smithi071.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:51:01.871 INFO:teuthology.orchestra.run.smithi071.stdout:Group : Unspecified 2024-03-29T10:51:01.871 INFO:teuthology.orchestra.run.smithi071.stdout:Size : 8264254 2024-03-29T10:51:01.871 INFO:teuthology.orchestra.run.smithi071.stdout:License : GPLv3+ 2024-03-29T10:51:01.871 INFO:teuthology.orchestra.run.smithi071.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:51:01.872 INFO:teuthology.orchestra.run.smithi071.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:51:01.872 INFO:teuthology.orchestra.run.smithi071.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:51:01.872 INFO:teuthology.orchestra.run.smithi071.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:51:01.872 INFO:teuthology.orchestra.run.smithi071.stdout:Packager : builder@centos.org 2024-03-29T10:51:01.872 INFO:teuthology.orchestra.run.smithi071.stdout:Vendor : CentOS 2024-03-29T10:51:01.872 INFO:teuthology.orchestra.run.smithi071.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:51:01.872 INFO:teuthology.orchestra.run.smithi071.stdout:Summary : Support tools for GRUB. 2024-03-29T10:51:01.872 INFO:teuthology.orchestra.run.smithi071.stdout:Description : 2024-03-29T10:51:01.872 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:51:01.872 INFO:teuthology.orchestra.run.smithi071.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:51:01.872 INFO:teuthology.orchestra.run.smithi071.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:51:01.872 INFO:teuthology.orchestra.run.smithi071.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:51:01.872 INFO:teuthology.orchestra.run.smithi071.stdout:hardware devices. 2024-03-29T10:51:01.873 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:51:01.873 INFO:teuthology.orchestra.run.smithi071.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:51:01.873 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:51:01.874 INFO:teuthology.task.kernel:Updating grub on smithi071 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:51:01.874 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:51:02.464 INFO:teuthology.orchestra.run.smithi071.stderr:Generating grub configuration file ... 2024-03-29T10:51:03.852 INFO:teuthology.orchestra.run.smithi071.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:51:03.913 INFO:teuthology.orchestra.run.smithi071.stderr:done 2024-03-29T10:51:03.916 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2024-03-29T10:51:03.932 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.Ed6dmypAui 2024-03-29T10:51:03.932 DEBUG:teuthology.orchestra.run.smithi071:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Ed6dmypAui 2024-03-29T10:51:04.002 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.Ed6dmypAui 2024-03-29T10:51:04.139 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.Ed6dmypAui is 6KB 2024-03-29T10:51:04.186 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.Ed6dmypAui 2024-03-29T10:51:04.200 DEBUG:teuthology.orchestra.run.smithi071:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:51:04.265 INFO:teuthology.orchestra.run.smithi071.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:51:04.266 INFO:teuthology.orchestra.run.smithi071.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:51:04.266 INFO:teuthology.orchestra.run.smithi071.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:51:04.266 INFO:teuthology.orchestra.run.smithi071.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:51:04.267 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:51:04.417 DEBUG:teuthology.orchestra.run.smithi071:> sudo shutdown -r now 2024-03-29T10:51:11.365 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:51:11.365 INFO:teuthology.orchestra.run.smithi137.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:51:11.365 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:51:11.408 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:51:11.787 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:51:11.788 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:51:11.788 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:51:11.788 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:51:12.187 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:51:12.188 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:51:12.188 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T10:51:12.188 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:51:12.188 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:51:12.188 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:51:12.188 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:51:12.188 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:51:12.188 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T10:51:12.521 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:51:12.615 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:51:12.615 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:51:12.616 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:51:12.616 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:51:12.616 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:51:12.616 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T10:51:12.616 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi137.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:51:12.616 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:51:12.696 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:51:12.696 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:51:12.696 DEBUG:teuthology.orchestra.run.smithi137:> sudo rpm -qi grub2-tools 2024-03-29T10:51:12.730 INFO:teuthology.orchestra.run.smithi137.stdout:Name : grub2-tools 2024-03-29T10:51:12.730 INFO:teuthology.orchestra.run.smithi137.stdout:Epoch : 1 2024-03-29T10:51:12.730 INFO:teuthology.orchestra.run.smithi137.stdout:Version : 2.06 2024-03-29T10:51:12.730 INFO:teuthology.orchestra.run.smithi137.stdout:Release : 68.el9 2024-03-29T10:51:12.730 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: x86_64 2024-03-29T10:51:12.730 INFO:teuthology.orchestra.run.smithi137.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:51:12.731 INFO:teuthology.orchestra.run.smithi137.stdout:Group : Unspecified 2024-03-29T10:51:12.731 INFO:teuthology.orchestra.run.smithi137.stdout:Size : 8264254 2024-03-29T10:51:12.731 INFO:teuthology.orchestra.run.smithi137.stdout:License : GPLv3+ 2024-03-29T10:51:12.731 INFO:teuthology.orchestra.run.smithi137.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:51:12.731 INFO:teuthology.orchestra.run.smithi137.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:51:12.731 INFO:teuthology.orchestra.run.smithi137.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:51:12.731 INFO:teuthology.orchestra.run.smithi137.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:51:12.731 INFO:teuthology.orchestra.run.smithi137.stdout:Packager : builder@centos.org 2024-03-29T10:51:12.731 INFO:teuthology.orchestra.run.smithi137.stdout:Vendor : CentOS 2024-03-29T10:51:12.731 INFO:teuthology.orchestra.run.smithi137.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:51:12.731 INFO:teuthology.orchestra.run.smithi137.stdout:Summary : Support tools for GRUB. 2024-03-29T10:51:12.731 INFO:teuthology.orchestra.run.smithi137.stdout:Description : 2024-03-29T10:51:12.731 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:51:12.731 INFO:teuthology.orchestra.run.smithi137.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:51:12.731 INFO:teuthology.orchestra.run.smithi137.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:51:12.732 INFO:teuthology.orchestra.run.smithi137.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:51:12.732 INFO:teuthology.orchestra.run.smithi137.stdout:hardware devices. 2024-03-29T10:51:12.732 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:51:12.732 INFO:teuthology.orchestra.run.smithi137.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:51:12.733 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:51:12.733 INFO:teuthology.task.kernel:Updating grub on smithi137 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:51:12.733 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:51:13.314 INFO:teuthology.orchestra.run.smithi137.stderr:Generating grub configuration file ... 2024-03-29T10:51:14.780 INFO:teuthology.orchestra.run.smithi137.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:51:14.812 INFO:teuthology.orchestra.run.smithi137.stderr:done 2024-03-29T10:51:14.815 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-03-29T10:51:14.830 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.V7rTR0p4qQ 2024-03-29T10:51:14.830 DEBUG:teuthology.orchestra.run.smithi137:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.V7rTR0p4qQ 2024-03-29T10:51:14.898 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.V7rTR0p4qQ 2024-03-29T10:51:15.044 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.V7rTR0p4qQ is 6KB 2024-03-29T10:51:15.093 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.V7rTR0p4qQ 2024-03-29T10:51:15.107 DEBUG:teuthology.orchestra.run.smithi137:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:51:15.171 INFO:teuthology.orchestra.run.smithi137.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:51:15.172 INFO:teuthology.orchestra.run.smithi137.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:51:15.172 INFO:teuthology.orchestra.run.smithi137.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:51:15.172 INFO:teuthology.orchestra.run.smithi137.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:51:15.173 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:51:15.317 DEBUG:teuthology.orchestra.run.smithi137:> sudo shutdown -r now 2024-03-29T10:51:34.430 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T10:51:34.431 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-29T10:51:34.431 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:51:45.330 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T10:51:45.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-03-29T10:51:45.331 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:51:52.990 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-03-29T10:52:01.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-29T10:52:01.999 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:52:03.742 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-03-29T10:52:05.054 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-03-29T10:52:12.747 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-03-29T10:52:12.748 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:52:15.806 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-03-29T10:52:17.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-29T10:52:17.056 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:52:27.814 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-03-29T10:52:27.815 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:52:48.862 DEBUG:teuthology.orchestra.run.smithi071:> true 2024-03-29T10:52:49.241 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-29T10:52:49.241 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:52:49.241 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2024-03-29T10:52:49.255 INFO:teuthology.orchestra.run.smithi071.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:52:49.256 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:52:49.256 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:52:49.256 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:52:50.257 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T10:52:50.257 DEBUG:teuthology.orchestra.run.smithi071:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:52:50.326 INFO:teuthology.orchestra.run.smithi071.stdout:ttyS1 2024-03-29T10:52:50.386 DEBUG:teuthology.parallel:result is None 2024-03-29T10:52:59.656 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-03-29T10:53:00.047 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-03-29T10:53:00.047 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:53:00.047 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-03-29T10:53:00.061 INFO:teuthology.orchestra.run.smithi137.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:53:00.061 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:53:00.061 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:53:00.061 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:53:01.063 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T10:53:01.063 DEBUG:teuthology.orchestra.run.smithi137:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:53:01.174 INFO:teuthology.orchestra.run.smithi137.stdout:ttyS1 2024-03-29T10:53:01.221 DEBUG:teuthology.parallel:result is None 2024-03-29T10:53:01.221 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T10:53:01.228 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T10:53:01.229 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:53:01.231 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:53:01.247 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T10:53:01.273 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T10:53:01.305 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T10:53:01.306 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:53:01.308 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:53:01.415 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T10:53:01.422 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T10:53:01.422 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:53:01.424 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:53:01.455 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:53:01.458 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:53:01.469 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:53:01.471 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:53:01.472 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T10:53:01.479 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T10:53:01.479 DEBUG:teuthology.orchestra.run.smithi071:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:53:01.514 DEBUG:teuthology.orchestra.run.smithi137:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:53:01.542 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T10:53:01.552 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T10:53:01.552 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:53:01.584 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:53:01.599 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:53:01.681 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:53:01.730 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:53:01.814 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:53:01.857 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T10:53:01.857 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:53:01.924 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:53:01.967 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:53:02.030 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:53:02.108 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:53:02.155 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T10:53:02.155 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:53:02.226 DEBUG:teuthology.orchestra.run.smithi071:> sudo service rsyslog restart 2024-03-29T10:53:02.243 DEBUG:teuthology.orchestra.run.smithi137:> sudo service rsyslog restart 2024-03-29T10:53:02.292 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:53:02.347 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:53:02.798 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T10:53:02.806 INFO:teuthology.task.internal:Starting timer... 2024-03-29T10:53:02.806 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T10:53:02.845 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T10:53:02.874 DEBUG:teuthology.orchestra.run.smithi071:> sudo service auditd rotate 2024-03-29T10:53:02.931 INFO:teuthology.orchestra.run.smithi071.stdout:Rotating logs: 2024-03-29T10:53:02.933 DEBUG:teuthology.orchestra.run.smithi137:> sudo service auditd rotate 2024-03-29T10:53:03.044 INFO:teuthology.orchestra.run.smithi137.stdout:Rotating logs: 2024-03-29T10:53:03.046 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T10:53:03.046 DEBUG:teuthology.orchestra.run.smithi071:> /usr/sbin/getenforce 2024-03-29T10:53:03.067 INFO:teuthology.orchestra.run.smithi071.stdout:Permissive 2024-03-29T10:53:03.067 DEBUG:teuthology.orchestra.run.smithi137:> /usr/sbin/getenforce 2024-03-29T10:53:03.091 INFO:teuthology.orchestra.run.smithi137.stdout:Permissive 2024-03-29T10:53:03.091 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi071.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi137.front.sepia.ceph.com': 'permissive'} 2024-03-29T10:53:03.092 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:53:03.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:53:03.136 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:53:03.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:53:03.163 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T10:53:03.163 DEBUG:teuthology.orchestra.run.smithi071:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:53:03.211 DEBUG:teuthology.orchestra.run.smithi137:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:53:03.461 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T10:53:03.470 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-29T10:53:03.476 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-29T10:53:03.632 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T10:53:03.659 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T10:53:03.660 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi071.front.sepia.ceph.com,smithi137.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T10:57:28.132 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi071.front.sepia.ceph.com'), Remote(name='ubuntu@smithi137.front.sepia.ceph.com')] 2024-03-29T10:57:28.133 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-29T10:57:28.135 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:57:28.226 DEBUG:teuthology.orchestra.run.smithi071:> true 2024-03-29T10:57:28.305 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-29T10:57:28.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-03-29T10:57:28.306 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:57:28.395 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-03-29T10:57:28.473 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-03-29T10:57:28.474 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T10:57:28.484 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T10:57:28.484 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:57:28.485 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:57:28.487 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:57:28.487 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:57:28.516 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:57:28.534 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:57:28.543 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:57:28.560 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:57:28.560 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: ntpd: command not found 2024-03-29T10:57:28.574 INFO:teuthology.orchestra.run.smithi071.stdout:506 Cannot talk to daemon 2024-03-29T10:57:28.589 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: ntpd: command not found 2024-03-29T10:57:28.590 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:57:28.602 INFO:teuthology.orchestra.run.smithi137.stdout:506 Cannot talk to daemon 2024-03-29T10:57:28.606 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:57:28.619 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:57:28.635 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:57:28.659 INFO:teuthology.orchestra.run.smithi071.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:57:28.662 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:57:28.662 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2024-03-29T10:57:28.662 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:57:28.662 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:57:28.663 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:57:28.663 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:57:28.687 INFO:teuthology.orchestra.run.smithi137.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:57:28.691 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:57:28.691 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2024-03-29T10:57:28.691 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:57:28.691 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:57:28.691 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:57:28.691 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:57:28.691 INFO:teuthology.run_tasks:Running task install... 2024-03-29T10:57:28.699 DEBUG:teuthology.task.install:project ceph 2024-03-29T10:57:28.699 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T10:57:28.699 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T10:57:28.699 INFO:teuthology.task.install:Using flavor: default 2024-03-29T10:57:28.704 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T10:57:28.704 INFO:teuthology.task.install:extra packages: [] 2024-03-29T10:57:28.705 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:57:28.705 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:57:28.707 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:57:28.707 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:57:28.857 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:57:28.857 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:57:29.007 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:57:29.007 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T10:57:29.007 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:57:29.035 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:57:29.035 DEBUG:teuthology.orchestra.run.smithi071:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:57:29.098 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:57:29.098 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:57:29.103 DEBUG:teuthology.orchestra.run.smithi071:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:57:29.184 DEBUG:teuthology.orchestra.run.smithi071:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:57:29.207 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:57:29.207 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T10:57:29.207 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:57:29.235 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:57:29.235 DEBUG:teuthology.orchestra.run.smithi137:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:57:29.250 INFO:teuthology.orchestra.run.smithi071.stdout:check_obsoletes = 1 2024-03-29T10:57:29.251 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum clean all 2024-03-29T10:57:29.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:57:29.385 DEBUG:teuthology.orchestra.run.smithi137:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:57:29.451 INFO:teuthology.orchestra.run.smithi137.stdout:check_obsoletes = 1 2024-03-29T10:57:29.452 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum clean all 2024-03-29T10:57:29.487 INFO:teuthology.orchestra.run.smithi071.stdout:74 files removed 2024-03-29T10:57:29.514 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-radosgw 2024-03-29T10:57:29.698 INFO:teuthology.orchestra.run.smithi137.stdout:74 files removed 2024-03-29T10:57:29.725 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-radosgw 2024-03-29T10:57:30.245 INFO:teuthology.orchestra.run.smithi071.stdout:ceph packages for x86_64 190 kB/s | 82 kB 00:00 2024-03-29T10:57:30.434 INFO:teuthology.orchestra.run.smithi137.stdout:ceph packages for x86_64 201 kB/s | 82 kB 00:00 2024-03-29T10:57:30.662 INFO:teuthology.orchestra.run.smithi071.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-29T10:57:30.818 INFO:teuthology.orchestra.run.smithi137.stdout:ceph noarch packages 30 kB/s | 10 kB 00:00 2024-03-29T10:57:31.088 INFO:teuthology.orchestra.run.smithi071.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-03-29T10:57:31.194 INFO:teuthology.orchestra.run.smithi137.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-03-29T10:57:31.474 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - AppStream 19 MB/s | 6.7 MB 00:00 2024-03-29T10:57:31.613 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-29T10:57:32.857 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - BaseOS 7.8 MB/s | 2.0 MB 00:00 2024-03-29T10:57:33.088 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2024-03-29T10:57:34.155 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-29T10:57:37.213 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - AppStream 9.5 MB/s | 19 MB 00:01 2024-03-29T10:57:41.114 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-29T10:57:42.213 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - BaseOS 911 kB/s | 8.0 MB 00:09 2024-03-29T10:57:42.907 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-29T10:57:44.114 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-03-29T10:57:45.621 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 8.0 MB/s | 21 MB 00:02 2024-03-29T10:57:48.539 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - CRB 6.6 MB/s | 6.1 MB 00:00 2024-03-29T10:57:50.065 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - Extras packages 85 kB/s | 16 kB 00:00 2024-03-29T10:57:50.558 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-03-29T10:57:51.008 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 25 MB/s | 21 MB 00:00 2024-03-29T10:57:53.226 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T10:57:53.230 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:57:53.230 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-29T10:57:53.230 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:57:53.230 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T10:57:53.230 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:57:53.230 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrading: 2024-03-29T10:57:53.230 INFO:teuthology.orchestra.run.smithi137.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:57:53.230 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:57:53.230 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-29T10:57:53.231 INFO:teuthology.orchestra.run.smithi137.stdout: boost-program-options 2024-03-29T10:57:53.231 INFO:teuthology.orchestra.run.smithi137.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:57:53.231 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:57:53.231 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:57:53.231 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:57:53.231 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:57:53.231 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:57:53.231 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:57:53.231 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:57:53.231 INFO:teuthology.orchestra.run.smithi137.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:57:53.231 INFO:teuthology.orchestra.run.smithi137.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:57:53.231 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:57:53.231 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:57:53.232 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:57:53.232 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:57:53.232 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:57:53.232 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:57:53.232 INFO:teuthology.orchestra.run.smithi137.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:57:53.232 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse 2024-03-29T10:57:53.232 INFO:teuthology.orchestra.run.smithi137.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:57:53.232 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:57:53.232 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:57:53.232 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:57:53.232 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:57:53.232 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:57:53.232 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:57:53.233 INFO:teuthology.orchestra.run.smithi137.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:57:53.233 INFO:teuthology.orchestra.run.smithi137.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:57:53.233 INFO:teuthology.orchestra.run.smithi137.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:57:53.233 INFO:teuthology.orchestra.run.smithi137.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:57:53.233 INFO:teuthology.orchestra.run.smithi137.stdout:Installing weak dependencies: 2024-03-29T10:57:53.233 INFO:teuthology.orchestra.run.smithi137.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:57:53.233 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:57:53.233 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T10:57:53.233 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:57:53.233 INFO:teuthology.orchestra.run.smithi137.stdout:Install 30 Packages 2024-03-29T10:57:53.233 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrade 2 Packages 2024-03-29T10:57:53.233 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:57:53.234 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 58 M 2024-03-29T10:57:53.234 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T10:57:53.944 INFO:teuthology.orchestra.run.smithi137.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 8.4 MB/s | 5.3 MB 00:00 2024-03-29T10:57:54.011 INFO:teuthology.orchestra.run.smithi137.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 377 kB/s | 25 kB 00:00 2024-03-29T10:57:54.111 INFO:teuthology.orchestra.run.smithi137.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 7.3 MB/s | 742 kB 00:00 2024-03-29T10:57:54.203 INFO:teuthology.orchestra.run.smithi137.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 5.3 MB/s | 498 kB 00:00 2024-03-29T10:57:54.687 INFO:teuthology.orchestra.run.smithi137.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 11 MB/s | 5.1 MB 00:00 2024-03-29T10:57:54.870 INFO:teuthology.orchestra.run.smithi137.stdout:(6/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 13 MB/s | 20 MB 00:01 2024-03-29T10:57:54.895 INFO:teuthology.orchestra.run.smithi137.stdout:(7/32): python3-ceph-argparse-19.0.0-1439.g5dcb 216 kB/s | 45 kB 00:00 2024-03-29T10:57:54.921 INFO:teuthology.orchestra.run.smithi137.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 2.6 MB/s | 131 kB 00:00 2024-03-29T10:57:54.946 INFO:teuthology.orchestra.run.smithi137.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 3.1 MB/s | 162 kB 00:00 2024-03-29T10:57:54.971 INFO:teuthology.orchestra.run.smithi137.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 6.3 MB/s | 320 kB 00:00 2024-03-29T10:57:54.997 INFO:teuthology.orchestra.run.smithi137.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 5.8 MB/s | 298 kB 00:00 2024-03-29T10:57:55.022 INFO:teuthology.orchestra.run.smithi137.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.9 MB/s | 99 kB 00:00 2024-03-29T10:57:55.164 INFO:teuthology.orchestra.run.smithi137.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 635 kB/s | 106 kB 00:00 2024-03-29T10:57:55.189 INFO:teuthology.orchestra.run.smithi137.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 955 kB/s | 159 kB 00:00 2024-03-29T10:57:55.214 INFO:teuthology.orchestra.run.smithi137.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 882 kB/s | 44 kB 00:00 2024-03-29T10:57:55.281 INFO:teuthology.orchestra.run.smithi137.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-03-29T10:57:55.306 INFO:teuthology.orchestra.run.smithi137.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-29T10:57:55.331 INFO:teuthology.orchestra.run.smithi137.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 828 kB/s | 41 kB 00:00 2024-03-29T10:57:55.357 INFO:teuthology.orchestra.run.smithi137.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-29T10:57:55.382 INFO:teuthology.orchestra.run.smithi137.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 698 kB/s | 35 kB 00:00 2024-03-29T10:57:55.407 INFO:teuthology.orchestra.run.smithi137.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 926 kB/s | 46 kB 00:00 2024-03-29T10:57:55.641 INFO:teuthology.orchestra.run.smithi137.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2024-03-29T10:57:55.691 INFO:teuthology.orchestra.run.smithi137.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 519 kB/s | 26 kB 00:00 2024-03-29T10:57:55.750 INFO:teuthology.orchestra.run.smithi137.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 833 kB/s | 49 kB 00:00 2024-03-29T10:57:55.817 INFO:teuthology.orchestra.run.smithi137.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-03-29T10:57:55.842 INFO:teuthology.orchestra.run.smithi137.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 732 kB/s | 67 kB 00:00 2024-03-29T10:57:55.884 INFO:teuthology.orchestra.run.smithi137.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-03-29T10:57:55.926 INFO:teuthology.orchestra.run.smithi137.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-03-29T10:57:55.968 INFO:teuthology.orchestra.run.smithi137.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-03-29T10:57:56.016 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-29T10:57:56.194 INFO:teuthology.orchestra.run.smithi137.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 13 MB/s | 3.4 MB 00:00 2024-03-29T10:57:56.277 INFO:teuthology.orchestra.run.smithi137.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 10 MB/s | 3.1 MB 00:00 2024-03-29T10:57:58.394 INFO:teuthology.orchestra.run.smithi137.stdout:(32/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 1.9 MB/s | 9.7 MB 00:05 2024-03-29T10:57:58.399 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:57:58.399 INFO:teuthology.orchestra.run.smithi137.stdout:Total 11 MB/s | 58 MB 00:05 2024-03-29T10:57:58.399 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T10:57:58.437 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T10:57:58.437 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T10:57:58.609 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T10:57:58.630 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:57:58.630 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-29T10:57:58.630 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:57:58.630 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T10:57:58.631 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:57:58.631 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrading: 2024-03-29T10:57:58.631 INFO:teuthology.orchestra.run.smithi071.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:57:58.631 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:57:58.631 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-29T10:57:58.631 INFO:teuthology.orchestra.run.smithi071.stdout: boost-program-options 2024-03-29T10:57:58.631 INFO:teuthology.orchestra.run.smithi071.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:57:58.631 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:57:58.631 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:57:58.631 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:57:58.631 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:57:58.631 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:57:58.632 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:57:58.632 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:57:58.632 INFO:teuthology.orchestra.run.smithi071.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:57:58.632 INFO:teuthology.orchestra.run.smithi071.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:57:58.632 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:57:58.632 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:57:58.632 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:57:58.632 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:57:58.632 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:57:58.632 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:57:58.632 INFO:teuthology.orchestra.run.smithi071.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:57:58.632 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse 2024-03-29T10:57:58.632 INFO:teuthology.orchestra.run.smithi071.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:57:58.632 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:57:58.633 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:57:58.633 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:57:58.633 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:57:58.633 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:57:58.633 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:57:58.633 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:57:58.633 INFO:teuthology.orchestra.run.smithi071.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:57:58.633 INFO:teuthology.orchestra.run.smithi071.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:57:58.633 INFO:teuthology.orchestra.run.smithi071.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:57:58.633 INFO:teuthology.orchestra.run.smithi071.stdout:Installing weak dependencies: 2024-03-29T10:57:58.633 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:57:58.633 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:57:58.633 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T10:57:58.633 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:57:58.634 INFO:teuthology.orchestra.run.smithi071.stdout:Install 30 Packages 2024-03-29T10:57:58.634 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrade 2 Packages 2024-03-29T10:57:58.634 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:57:58.634 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 58 M 2024-03-29T10:57:58.634 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T10:57:58.907 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T10:57:58.908 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T10:58:00.017 INFO:teuthology.orchestra.run.smithi071.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 4.3 MB/s | 5.3 MB 00:01 2024-03-29T10:58:00.067 INFO:teuthology.orchestra.run.smithi071.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 507 kB/s | 25 kB 00:00 2024-03-29T10:58:00.200 INFO:teuthology.orchestra.run.smithi071.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 5.4 MB/s | 742 kB 00:00 2024-03-29T10:58:00.301 INFO:teuthology.orchestra.run.smithi071.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 4.9 MB/s | 498 kB 00:00 2024-03-29T10:58:00.369 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T10:58:00.448 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:58:00.482 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:58:00.507 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:58:00.538 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:58:00.555 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:58:00.747 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:58:00.802 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:58:01.009 INFO:teuthology.orchestra.run.smithi071.stdout:(5/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 8.9 MB/s | 20 MB 00:02 2024-03-29T10:58:01.093 INFO:teuthology.orchestra.run.smithi071.stdout:(6/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 6.4 MB/s | 5.1 MB 00:00 2024-03-29T10:58:01.118 INFO:teuthology.orchestra.run.smithi071.stdout:(7/32): python3-ceph-argparse-19.0.0-1439.g5dcb 416 kB/s | 45 kB 00:00 2024-03-29T10:58:01.132 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:58:01.151 INFO:teuthology.orchestra.run.smithi071.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 2.2 MB/s | 131 kB 00:00 2024-03-29T10:58:01.177 INFO:teuthology.orchestra.run.smithi071.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.7 MB/s | 162 kB 00:00 2024-03-29T10:58:01.192 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:58:01.211 INFO:teuthology.orchestra.run.smithi071.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 5.3 MB/s | 320 kB 00:00 2024-03-29T10:58:01.236 INFO:teuthology.orchestra.run.smithi071.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 4.9 MB/s | 298 kB 00:00 2024-03-29T10:58:01.261 INFO:teuthology.orchestra.run.smithi071.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.9 MB/s | 99 kB 00:00 2024-03-29T10:58:01.284 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:58:01.375 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:58:01.402 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:58:01.428 INFO:teuthology.orchestra.run.smithi071.stdout:(13/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 3.7 MB/s | 9.7 MB 00:02 2024-03-29T10:58:01.448 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:58:01.470 INFO:teuthology.orchestra.run.smithi071.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 453 kB/s | 106 kB 00:00 2024-03-29T10:58:01.477 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:58:01.503 INFO:teuthology.orchestra.run.smithi071.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 601 kB/s | 44 kB 00:00 2024-03-29T10:58:01.521 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:58:01.554 INFO:teuthology.orchestra.run.smithi071.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 594 kB/s | 159 kB 00:00 2024-03-29T10:58:01.590 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:58:01.612 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:58:01.659 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:58:01.694 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:58:01.804 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:58:01.812 INFO:teuthology.orchestra.run.smithi071.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 1.9 MB/s | 661 kB 00:00 2024-03-29T10:58:01.837 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:58:01.878 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:58:01.888 INFO:teuthology.orchestra.run.smithi071.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 736 kB/s | 282 kB 00:00 2024-03-29T10:58:01.910 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:58:01.913 INFO:teuthology.orchestra.run.smithi071.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 115 kB/s | 41 kB 00:00 2024-03-29T10:58:01.939 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:58:01.946 INFO:teuthology.orchestra.run.smithi071.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.6 MB/s | 220 kB 00:00 2024-03-29T10:58:01.972 INFO:teuthology.orchestra.run.smithi071.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 416 kB/s | 35 kB 00:00 2024-03-29T10:58:01.972 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:58:01.988 INFO:teuthology.orchestra.run.smithi071.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 615 kB/s | 46 kB 00:00 2024-03-29T10:58:02.295 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:58:02.305 INFO:teuthology.orchestra.run.smithi071.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 82 kB/s | 26 kB 00:00 2024-03-29T10:58:02.335 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:58:02.408 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:58:02.431 INFO:teuthology.orchestra.run.smithi071.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 638 kB/s | 309 kB 00:00 2024-03-29T10:58:02.440 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:58:02.447 INFO:teuthology.orchestra.run.smithi071.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 343 kB/s | 49 kB 00:00 2024-03-29T10:58:02.483 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:58:02.506 INFO:teuthology.orchestra.run.smithi071.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 897 kB/s | 67 kB 00:00 2024-03-29T10:58:02.606 INFO:teuthology.orchestra.run.smithi071.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-29T10:58:02.715 INFO:teuthology.orchestra.run.smithi071.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 6.1 MB/s | 4.4 MB 00:00 2024-03-29T10:58:02.757 INFO:teuthology.orchestra.run.smithi071.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.6 MB/s | 838 kB 00:00 2024-03-29T10:58:02.874 INFO:teuthology.orchestra.run.smithi071.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 5.9 MB/s | 1.6 MB 00:00 2024-03-29T10:58:02.930 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:58:02.962 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:58:03.034 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:58:03.101 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:58:03.131 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:58:03.216 INFO:teuthology.orchestra.run.smithi071.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 6.7 MB/s | 3.4 MB 00:00 2024-03-29T10:58:03.732 INFO:teuthology.orchestra.run.smithi071.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 3.2 MB/s | 3.1 MB 00:00 2024-03-29T10:58:03.740 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:58:03.740 INFO:teuthology.orchestra.run.smithi071.stdout:Total 11 MB/s | 58 MB 00:05 2024-03-29T10:58:03.741 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T10:58:03.778 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T10:58:03.779 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T10:58:04.260 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T10:58:04.260 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T10:58:05.350 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:58:05.395 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:58:05.695 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T10:58:05.782 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:58:05.875 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:58:05.946 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:58:05.954 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:58:05.977 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:58:05.989 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:58:05.994 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:58:06.063 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:58:06.063 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:58:06.063 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:58:06.063 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:06.116 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:58:06.136 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:58:06.199 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:58:06.532 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:58:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:58:06.681 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:58:06.770 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:58:06.799 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:58:06.845 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:58:06.874 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:58:06.918 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:58:06.986 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:58:07.009 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:58:07.098 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:58:07.133 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:58:07.243 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:58:07.284 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:58:07.333 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:58:07.374 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:58:07.403 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:58:07.435 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:58:07.761 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:58:07.790 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:58:07.863 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:58:07.895 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:58:07.939 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:58:08.397 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:58:08.426 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:58:08.498 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:58:08.565 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:58:08.595 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:58:10.711 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:58:10.749 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:58:11.301 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:58:11.336 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:58:11.428 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:58:11.428 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:58:11.428 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:58:11.428 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:58:11.471 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:58:15.767 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:58:15.767 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /sys 2024-03-29T10:58:15.767 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /proc 2024-03-29T10:58:15.767 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /mnt 2024-03-29T10:58:15.767 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /var/tmp 2024-03-29T10:58:15.767 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /home 2024-03-29T10:58:15.768 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /root 2024-03-29T10:58:15.768 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /tmp 2024-03-29T10:58:15.768 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:16.686 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:58:16.711 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:58:16.711 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:58:16.711 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:58:16.711 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:58:16.711 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:58:16.711 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:16.713 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:58:16.780 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:58:16.780 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:58:17.908 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:58:17.908 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:58:17.908 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:58:17.908 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:58:17.908 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:58:17.908 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:58:17.908 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:58:17.908 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:58:17.908 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:58:17.908 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:58:17.908 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:58:17.908 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:58:17.909 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:58:17.909 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:58:17.909 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:58:17.909 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:58:17.909 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:58:17.909 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:58:17.909 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:58:17.909 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:58:17.909 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:58:17.909 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:58:17.909 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:58:17.909 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:58:17.909 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:58:17.910 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:58:17.910 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:58:17.910 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:58:17.910 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:58:17.910 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:58:17.910 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:58:17.910 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:58:17.910 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:58:17.910 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:58:18.897 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:58:18.897 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:18.897 INFO:teuthology.orchestra.run.smithi137.stdout:Upgraded: 2024-03-29T10:58:18.898 INFO:teuthology.orchestra.run.smithi137.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:18.898 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:18.898 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T10:58:18.898 INFO:teuthology.orchestra.run.smithi137.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:58:18.898 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:18.898 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:18.898 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:18.898 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:18.898 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:58:18.898 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:58:18.898 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:58:18.898 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:18.898 INFO:teuthology.orchestra.run.smithi137.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:58:18.899 INFO:teuthology.orchestra.run.smithi137.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:58:18.899 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:58:18.899 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:18.899 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:58:18.899 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:18.899 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:58:18.899 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:58:18.899 INFO:teuthology.orchestra.run.smithi137.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:58:18.899 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:18.899 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:18.899 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:18.899 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:58:18.899 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:18.899 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:18.899 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:18.900 INFO:teuthology.orchestra.run.smithi137.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:58:18.900 INFO:teuthology.orchestra.run.smithi137.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:58:18.900 INFO:teuthology.orchestra.run.smithi137.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:58:18.900 INFO:teuthology.orchestra.run.smithi137.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:58:18.900 INFO:teuthology.orchestra.run.smithi137.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:58:18.900 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:18.900 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T10:58:19.201 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-test 2024-03-29T10:58:19.755 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:29 ago on Fri 29 Mar 2024 10:57:50 AM UTC. 2024-03-29T10:58:19.909 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T10:58:19.910 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:58:19.910 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-29T10:58:19.910 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:58:19.913 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T10:58:19.913 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:58:19.913 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-29T10:58:19.913 INFO:teuthology.orchestra.run.smithi137.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:58:19.913 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:58:19.914 INFO:teuthology.orchestra.run.smithi137.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:58:19.914 INFO:teuthology.orchestra.run.smithi137.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:58:19.914 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:58:19.914 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:19.914 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T10:58:19.914 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:58:19.914 INFO:teuthology.orchestra.run.smithi137.stdout:Install 6 Packages 2024-03-29T10:58:19.914 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:19.915 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 48 M 2024-03-29T10:58:19.915 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 204 M 2024-03-29T10:58:19.915 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T10:58:20.125 INFO:teuthology.orchestra.run.smithi137.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-29T10:58:20.167 INFO:teuthology.orchestra.run.smithi137.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2024-03-29T10:58:20.193 INFO:teuthology.orchestra.run.smithi137.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2024-03-29T10:58:20.335 INFO:teuthology.orchestra.run.smithi137.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 447 kB/s | 163 kB 00:00 2024-03-29T10:58:20.435 INFO:teuthology.orchestra.run.smithi137.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 775 kB/s | 187 kB 00:00 2024-03-29T10:58:21.243 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:58:21.277 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /sys 2024-03-29T10:58:21.278 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /proc 2024-03-29T10:58:21.278 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /mnt 2024-03-29T10:58:21.278 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /var/tmp 2024-03-29T10:58:21.278 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /home 2024-03-29T10:58:21.278 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /root 2024-03-29T10:58:21.278 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /tmp 2024-03-29T10:58:21.278 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:58:22.164 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:58:22.189 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:58:22.189 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:58:22.189 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:58:22.189 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:58:22.189 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:58:22.189 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:58:22.192 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:58:22.275 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:58:22.275 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:58:22.293 INFO:teuthology.orchestra.run.smithi137.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 20 MB/s | 47 MB 00:02 2024-03-29T10:58:22.297 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:58:22.297 INFO:teuthology.orchestra.run.smithi137.stdout:Total 20 MB/s | 48 MB 00:02 2024-03-29T10:58:22.311 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T10:58:22.332 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T10:58:22.332 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T10:58:22.712 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T10:58:22.712 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T10:58:23.288 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T10:58:23.352 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:58:23.368 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:58:23.368 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:58:23.368 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:58:23.368 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:58:23.369 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:58:23.369 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:58:23.369 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:58:23.369 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:58:23.369 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:58:23.369 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:58:23.369 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:58:23.369 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:58:23.369 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:58:23.369 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:58:23.369 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:58:23.369 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:58:23.369 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:58:23.369 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:58:23.370 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:58:23.370 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:58:23.370 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:58:23.370 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:58:23.370 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:58:23.370 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:58:23.370 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:58:23.370 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:58:23.370 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:58:23.370 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:58:23.371 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:58:23.371 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:58:23.371 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:58:23.371 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:58:23.371 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:58:23.371 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:58:23.403 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:58:23.444 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:58:23.494 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:58:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:58:24.530 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:58:24.530 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:58:24.530 INFO:teuthology.orchestra.run.smithi071.stdout:Upgraded: 2024-03-29T10:58:24.530 INFO:teuthology.orchestra.run.smithi071.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:24.530 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:24.530 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T10:58:24.530 INFO:teuthology.orchestra.run.smithi071.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:58:24.530 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:24.530 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:24.531 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:24.531 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:24.531 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:58:24.531 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:58:24.531 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:58:24.531 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:24.531 INFO:teuthology.orchestra.run.smithi071.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:58:24.531 INFO:teuthology.orchestra.run.smithi071.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:58:24.531 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:58:24.531 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:24.531 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:58:24.531 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:24.531 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:58:24.531 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:58:24.532 INFO:teuthology.orchestra.run.smithi071.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:58:24.532 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:24.532 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:24.532 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:24.532 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:58:24.532 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:24.532 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:24.532 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:24.532 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:58:24.532 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:58:24.532 INFO:teuthology.orchestra.run.smithi071.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:58:24.532 INFO:teuthology.orchestra.run.smithi071.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:58:24.532 INFO:teuthology.orchestra.run.smithi071.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:58:24.532 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:58:24.533 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T10:58:24.844 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-test 2024-03-29T10:58:25.408 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:29 ago on Fri 29 Mar 2024 10:57:56 AM UTC. 2024-03-29T10:58:25.567 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T10:58:25.567 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:58:25.567 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-29T10:58:25.567 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:58:25.568 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T10:58:25.568 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:58:25.568 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-29T10:58:25.568 INFO:teuthology.orchestra.run.smithi071.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:58:25.568 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:58:25.568 INFO:teuthology.orchestra.run.smithi071.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:58:25.568 INFO:teuthology.orchestra.run.smithi071.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:58:25.568 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:58:25.568 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:58:25.568 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T10:58:25.568 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:58:25.568 INFO:teuthology.orchestra.run.smithi071.stdout:Install 6 Packages 2024-03-29T10:58:25.569 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:58:25.569 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 48 M 2024-03-29T10:58:25.569 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 204 M 2024-03-29T10:58:25.569 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T10:58:25.881 INFO:teuthology.orchestra.run.smithi071.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T10:58:25.947 INFO:teuthology.orchestra.run.smithi071.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 300 kB 00:00 2024-03-29T10:58:25.971 INFO:teuthology.orchestra.run.smithi071.stdout:(3/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 677 kB/s | 163 kB 00:00 2024-03-29T10:58:25.989 INFO:teuthology.orchestra.run.smithi071.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-29T10:58:26.864 INFO:teuthology.orchestra.run.smithi071.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 208 kB/s | 187 kB 00:00 2024-03-29T10:58:28.001 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:58:28.071 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:58:28.190 INFO:teuthology.orchestra.run.smithi071.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 19 MB/s | 47 MB 00:02 2024-03-29T10:58:28.192 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:58:28.193 INFO:teuthology.orchestra.run.smithi071.stdout:Total 18 MB/s | 48 MB 00:02 2024-03-29T10:58:28.207 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T10:58:28.228 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T10:58:28.229 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T10:58:28.612 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T10:58:28.612 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T10:58:29.172 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T10:58:29.231 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:58:29.275 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:58:29.315 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:58:29.356 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:58:29.384 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:58:30.324 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:58:30.325 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:58:30.325 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:58:30.325 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:58:30.325 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:58:30.325 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:58:30.722 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:58:30.723 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:30.723 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T10:58:30.723 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:30.723 INFO:teuthology.orchestra.run.smithi137.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:58:30.723 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:30.723 INFO:teuthology.orchestra.run.smithi137.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:58:30.723 INFO:teuthology.orchestra.run.smithi137.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:58:30.724 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:58:30.724 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:30.724 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T10:58:30.987 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph 2024-03-29T10:58:31.568 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:41 ago on Fri 29 Mar 2024 10:57:50 AM UTC. 2024-03-29T10:58:31.736 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T10:58:31.739 INFO:teuthology.orchestra.run.smithi137.stdout:========================================================================================= 2024-03-29T10:58:31.739 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-29T10:58:31.739 INFO:teuthology.orchestra.run.smithi137.stdout:========================================================================================= 2024-03-29T10:58:31.739 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T10:58:31.739 INFO:teuthology.orchestra.run.smithi137.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:58:31.739 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-29T10:58:31.739 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:58:31.739 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:58:31.740 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:58:31.740 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:58:31.740 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:58:31.740 INFO:teuthology.orchestra.run.smithi137.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:58:31.740 INFO:teuthology.orchestra.run.smithi137.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:58:31.740 INFO:teuthology.orchestra.run.smithi137.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:58:31.740 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:58:31.740 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:58:31.740 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:58:31.740 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:58:31.740 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:58:31.740 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:58:31.740 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:58:31.740 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:58:31.741 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:58:31.741 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:58:31.741 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:58:31.741 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:58:31.741 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:58:31.741 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:58:31.741 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:58:31.741 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:58:31.741 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:58:31.741 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:58:31.741 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:58:31.741 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:58:31.741 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:58:31.742 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:58:31.742 INFO:teuthology.orchestra.run.smithi137.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:58:31.742 INFO:teuthology.orchestra.run.smithi137.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:58:31.742 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:58:31.742 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:58:31.742 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:58:31.742 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:31.742 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T10:58:31.742 INFO:teuthology.orchestra.run.smithi137.stdout:========================================================================================= 2024-03-29T10:58:31.742 INFO:teuthology.orchestra.run.smithi137.stdout:Install 36 Packages 2024-03-29T10:58:31.742 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:31.743 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 30 M 2024-03-29T10:58:31.743 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 107 M 2024-03-29T10:58:31.743 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T10:58:32.196 INFO:teuthology.orchestra.run.smithi137.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 39 kB/s | 6.6 kB 00:00 2024-03-29T10:58:32.688 INFO:teuthology.orchestra.run.smithi137.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 2.2 MB/s | 1.5 MB 00:00 2024-03-29T10:58:32.838 INFO:teuthology.orchestra.run.smithi137.stdout:(3/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 7.3 MB/s | 4.7 MB 00:00 2024-03-29T10:58:32.880 INFO:teuthology.orchestra.run.smithi137.stdout:(4/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 2.5 MB/s | 2.1 MB 00:00 2024-03-29T10:58:32.905 INFO:teuthology.orchestra.run.smithi137.stdout:(5/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 3.6 MB/s | 247 kB 00:00 2024-03-29T10:58:33.031 INFO:teuthology.orchestra.run.smithi137.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-29T10:58:33.056 INFO:teuthology.orchestra.run.smithi137.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2024-03-29T10:58:33.082 INFO:teuthology.orchestra.run.smithi137.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 631 kB/s | 32 kB 00:00 2024-03-29T10:58:33.106 INFO:teuthology.orchestra.run.smithi137.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 941 kB/s | 47 kB 00:00 2024-03-29T10:58:33.132 INFO:teuthology.orchestra.run.smithi137.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 773 kB/s | 39 kB 00:00 2024-03-29T10:58:33.157 INFO:teuthology.orchestra.run.smithi137.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-29T10:58:33.216 INFO:teuthology.orchestra.run.smithi137.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-29T10:58:33.241 INFO:teuthology.orchestra.run.smithi137.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-29T10:58:33.258 INFO:teuthology.orchestra.run.smithi137.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-29T10:58:33.283 INFO:teuthology.orchestra.run.smithi137.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-29T10:58:33.400 INFO:teuthology.orchestra.run.smithi137.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 191 kB/s | 22 kB 00:00 2024-03-29T10:58:33.841 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:58:33.875 INFO:teuthology.orchestra.run.smithi137.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 318 kB/s | 151 kB 00:00 2024-03-29T10:58:33.916 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:58:33.984 INFO:teuthology.orchestra.run.smithi137.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 402 kB/s | 43 kB 00:00 2024-03-29T10:58:34.034 INFO:teuthology.orchestra.run.smithi137.stdout:(19/36): python3-requests-2.25.1-8.el9.noarch.r 161 kB/s | 125 kB 00:00 2024-03-29T10:58:34.284 INFO:teuthology.orchestra.run.smithi137.stdout:(20/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 11 MB/s | 17 MB 00:01 2024-03-29T10:58:34.309 INFO:teuthology.orchestra.run.smithi137.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 527 kB/s | 172 kB 00:00 2024-03-29T10:58:34.410 INFO:teuthology.orchestra.run.smithi137.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 86 kB/s | 11 kB 00:00 2024-03-29T10:58:34.510 INFO:teuthology.orchestra.run.smithi137.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-03-29T10:58:34.579 INFO:teuthology.orchestra.run.smithi137.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 66 kB/s | 18 kB 00:00 2024-03-29T10:58:34.613 INFO:teuthology.orchestra.run.smithi137.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 190 kB/s | 19 kB 00:00 2024-03-29T10:58:34.705 INFO:teuthology.orchestra.run.smithi137.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 157 kB/s | 20 kB 00:00 2024-03-29T10:58:34.730 INFO:teuthology.orchestra.run.smithi137.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 396 kB/s | 46 kB 00:00 2024-03-29T10:58:34.780 INFO:teuthology.orchestra.run.smithi137.stdout:(28/36): python3-cherrypy-18.6.1-2.el9.noarch.r 480 kB/s | 358 kB 00:00 2024-03-29T10:58:34.881 INFO:teuthology.orchestra.run.smithi137.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-03-29T10:58:34.906 INFO:teuthology.orchestra.run.smithi137.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-03-29T10:58:34.948 INFO:teuthology.orchestra.run.smithi137.stdout:(31/36): python3-more-itertools-8.12.0-2.el9.no 325 kB/s | 79 kB 00:00 2024-03-29T10:58:34.990 INFO:teuthology.orchestra.run.smithi137.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 826 kB/s | 90 kB 00:00 2024-03-29T10:58:35.015 INFO:teuthology.orchestra.run.smithi137.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 329 kB/s | 36 kB 00:00 2024-03-29T10:58:35.115 INFO:teuthology.orchestra.run.smithi137.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 200 kB/s | 20 kB 00:00 2024-03-29T10:58:35.149 INFO:teuthology.orchestra.run.smithi137.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.6 MB/s | 427 kB 00:00 2024-03-29T10:58:35.174 INFO:teuthology.orchestra.run.smithi137.stdout:(36/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.0 MB/s | 230 kB 00:00 2024-03-29T10:58:35.178 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:58:35.178 INFO:teuthology.orchestra.run.smithi137.stdout:Total 8.6 MB/s | 30 MB 00:03 2024-03-29T10:58:35.203 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T10:58:35.245 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T10:58:35.245 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T10:58:35.677 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T10:58:35.677 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T10:58:36.222 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:58:36.223 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:58:36.223 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:58:36.223 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:58:36.223 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:58:36.223 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:58:36.598 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:58:36.598 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:58:36.598 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T10:58:36.599 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:36.599 INFO:teuthology.orchestra.run.smithi071.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:58:36.599 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:36.599 INFO:teuthology.orchestra.run.smithi071.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:58:36.599 INFO:teuthology.orchestra.run.smithi071.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:58:36.599 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:58:36.599 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:58:36.599 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T10:58:36.918 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph 2024-03-29T10:58:37.033 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T10:58:37.093 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:58:37.130 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:58:37.162 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:58:37.234 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:58:37.278 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:58:37.311 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:58:37.402 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:58:37.483 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:58:37.511 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:41 ago on Fri 29 Mar 2024 10:57:56 AM UTC. 2024-03-29T10:58:37.521 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:58:37.594 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:58:37.648 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:58:37.672 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T10:58:37.674 INFO:teuthology.orchestra.run.smithi071.stdout:========================================================================================= 2024-03-29T10:58:37.674 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-29T10:58:37.674 INFO:teuthology.orchestra.run.smithi071.stdout:========================================================================================= 2024-03-29T10:58:37.675 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T10:58:37.675 INFO:teuthology.orchestra.run.smithi071.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:58:37.675 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-29T10:58:37.675 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:58:37.675 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:58:37.675 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:58:37.675 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:58:37.675 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:58:37.675 INFO:teuthology.orchestra.run.smithi071.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:58:37.675 INFO:teuthology.orchestra.run.smithi071.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:58:37.675 INFO:teuthology.orchestra.run.smithi071.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:58:37.675 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:58:37.676 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:58:37.676 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:58:37.676 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:58:37.676 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:58:37.676 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:58:37.676 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:58:37.676 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:58:37.676 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:58:37.676 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:58:37.676 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:58:37.676 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:58:37.676 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:58:37.676 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:58:37.676 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:58:37.677 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:58:37.677 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:58:37.677 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:58:37.677 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:58:37.677 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:58:37.677 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:58:37.677 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:58:37.677 INFO:teuthology.orchestra.run.smithi071.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:58:37.677 INFO:teuthology.orchestra.run.smithi071.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:58:37.677 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:58:37.677 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:58:37.677 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:58:37.677 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:58:37.678 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T10:58:37.678 INFO:teuthology.orchestra.run.smithi071.stdout:========================================================================================= 2024-03-29T10:58:37.678 INFO:teuthology.orchestra.run.smithi071.stdout:Install 36 Packages 2024-03-29T10:58:37.678 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:58:37.678 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 30 M 2024-03-29T10:58:37.678 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 107 M 2024-03-29T10:58:37.678 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T10:58:37.686 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:58:37.735 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:58:37.789 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:58:37.975 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:58:38.032 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:58:38.070 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:58:38.105 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:58:38.135 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:58:38.158 INFO:teuthology.orchestra.run.smithi071.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 29 kB/s | 6.6 kB 00:00 2024-03-29T10:58:38.211 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:58:38.245 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:58:38.278 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:58:38.319 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:58:38.354 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:58:38.397 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:58:38.442 INFO:teuthology.orchestra.run.smithi071.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 4.1 MB/s | 2.1 MB 00:00 2024-03-29T10:58:38.486 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:58:38.527 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:58:38.599 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:58:38.633 INFO:teuthology.orchestra.run.smithi071.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 2.1 MB/s | 1.5 MB 00:00 2024-03-29T10:58:38.737 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:58:38.742 INFO:teuthology.orchestra.run.smithi071.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.2 MB/s | 247 kB 00:00 2024-03-29T10:58:38.851 INFO:teuthology.orchestra.run.smithi071.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-03-29T10:58:38.864 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:58:38.893 INFO:teuthology.orchestra.run.smithi071.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T10:58:38.926 INFO:teuthology.orchestra.run.smithi071.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 953 kB/s | 32 kB 00:00 2024-03-29T10:58:38.959 INFO:teuthology.orchestra.run.smithi071.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T10:58:39.004 INFO:teuthology.orchestra.run.smithi071.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 876 kB/s | 39 kB 00:00 2024-03-29T10:58:39.028 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:58:39.045 INFO:teuthology.orchestra.run.smithi071.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-29T10:58:39.057 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:58:39.082 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:58:39.082 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:58:39.083 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:58:39.083 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:58:39.083 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:58:39.083 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:39.113 INFO:teuthology.orchestra.run.smithi071.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T10:58:39.180 INFO:teuthology.orchestra.run.smithi071.stdout:(12/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 4.6 MB/s | 4.7 MB 00:01 2024-03-29T10:58:39.205 INFO:teuthology.orchestra.run.smithi071.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-29T10:58:39.230 INFO:teuthology.orchestra.run.smithi071.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T10:58:39.264 INFO:teuthology.orchestra.run.smithi071.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2024-03-29T10:58:39.339 INFO:teuthology.orchestra.run.smithi071.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 297 kB/s | 22 kB 00:00 2024-03-29T10:58:39.548 INFO:teuthology.orchestra.run.smithi071.stdout:(17/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 15 MB/s | 17 MB 00:01 2024-03-29T10:58:39.606 INFO:teuthology.orchestra.run.smithi071.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 333 kB/s | 125 kB 00:00 2024-03-29T10:58:39.648 INFO:teuthology.orchestra.run.smithi071.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 435 kB/s | 43 kB 00:00 2024-03-29T10:58:39.773 INFO:teuthology.orchestra.run.smithi071.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-03-29T10:58:39.799 INFO:teuthology.orchestra.run.smithi071.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 894 kB/s | 172 kB 00:00 2024-03-29T10:58:39.824 INFO:teuthology.orchestra.run.smithi071.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 212 kB/s | 11 kB 00:00 2024-03-29T10:58:39.849 INFO:teuthology.orchestra.run.smithi071.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 351 kB/s | 18 kB 00:00 2024-03-29T10:58:39.874 INFO:teuthology.orchestra.run.smithi071.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 463 kB/s | 23 kB 00:00 2024-03-29T10:58:39.900 INFO:teuthology.orchestra.run.smithi071.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-03-29T10:58:39.925 INFO:teuthology.orchestra.run.smithi071.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-03-29T10:58:39.950 INFO:teuthology.orchestra.run.smithi071.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-03-29T10:58:39.975 INFO:teuthology.orchestra.run.smithi071.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-03-29T10:58:40.009 INFO:teuthology.orchestra.run.smithi071.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2024-03-29T10:58:40.034 INFO:teuthology.orchestra.run.smithi071.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 279 kB/s | 16 kB 00:00 2024-03-29T10:58:40.060 INFO:teuthology.orchestra.run.smithi071.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-03-29T10:58:40.085 INFO:teuthology.orchestra.run.smithi071.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 716 kB/s | 36 kB 00:00 2024-03-29T10:58:40.110 INFO:teuthology.orchestra.run.smithi071.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-29T10:58:40.152 INFO:teuthology.orchestra.run.smithi071.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 480 kB/s | 20 kB 00:00 2024-03-29T10:58:40.177 INFO:teuthology.orchestra.run.smithi071.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-03-29T10:58:40.508 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:58:40.534 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:58:40.534 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:58:40.535 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:58:40.535 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:58:40.535 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:58:40.535 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:40.975 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:58:40.998 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:58:40.999 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:58:40.999 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:58:40.999 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:58:40.999 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:58:40.999 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:41.201 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:58:41.224 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:58:41.224 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:58:41.224 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:58:41.224 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:58:41.224 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:58:41.224 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:41.290 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:58:42.979 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:58:42.979 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:58:42.979 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:58:42.979 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:58:42.979 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:58:42.979 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:58:42.979 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:58:42.979 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:58:42.979 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:58:42.979 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:58:42.979 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:58:42.979 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:58:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:58:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:58:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:58:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:58:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:58:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:58:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:58:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:58:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:58:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:58:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:58:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:58:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:58:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:58:42.982 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:58:42.982 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:58:42.982 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:58:42.982 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:58:42.982 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:58:42.982 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:58:42.982 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:58:42.982 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:58:42.982 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:58:42.982 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:58:43.619 INFO:teuthology.orchestra.run.smithi071.stdout:(36/36): luarocks-3.9.2-1.el9.noarch.rpm 35 kB/s | 151 kB 00:04 2024-03-29T10:58:43.622 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:58:43.622 INFO:teuthology.orchestra.run.smithi071.stdout:Total 5.0 MB/s | 30 MB 00:05 2024-03-29T10:58:43.646 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T10:58:43.688 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T10:58:43.688 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T10:58:44.066 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:58:44.066 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:44.066 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T10:58:44.066 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:44.067 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:44.067 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:44.067 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:58:44.067 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:44.067 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:44.067 INFO:teuthology.orchestra.run.smithi137.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:58:44.067 INFO:teuthology.orchestra.run.smithi137.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:58:44.067 INFO:teuthology.orchestra.run.smithi137.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:58:44.067 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:58:44.067 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:58:44.067 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:58:44.067 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:58:44.067 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:58:44.068 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:58:44.068 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:58:44.068 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:58:44.068 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:58:44.068 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:58:44.068 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:58:44.068 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:58:44.068 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:58:44.068 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:58:44.068 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:58:44.068 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:58:44.068 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:58:44.068 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:58:44.069 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:58:44.069 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:58:44.069 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:58:44.069 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:58:44.069 INFO:teuthology.orchestra.run.smithi137.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:58:44.069 INFO:teuthology.orchestra.run.smithi137.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:58:44.069 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:58:44.069 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:58:44.069 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:58:44.069 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:44.069 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T10:58:44.078 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T10:58:44.078 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T10:58:44.403 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-base 2024-03-29T10:58:44.965 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:54 ago on Fri 29 Mar 2024 10:57:50 AM UTC. 2024-03-29T10:58:45.063 INFO:teuthology.orchestra.run.smithi137.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:58:45.114 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T10:58:45.115 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-03-29T10:58:45.115 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T10:58:45.186 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install cephadm 2024-03-29T10:58:45.396 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T10:58:45.454 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:58:45.490 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:58:45.523 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:58:45.592 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:58:45.630 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:58:45.681 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:58:45.751 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:55 ago on Fri 29 Mar 2024 10:57:50 AM UTC. 2024-03-29T10:58:45.770 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:58:45.849 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:58:45.891 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:58:45.902 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T10:58:45.903 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:58:45.903 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-29T10:58:45.903 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:58:45.903 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T10:58:45.903 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:58:45.903 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:45.903 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T10:58:45.903 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:58:45.903 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-03-29T10:58:45.904 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:45.904 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 747 k 2024-03-29T10:58:45.904 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 754 k 2024-03-29T10:58:45.904 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T10:58:45.963 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:58:46.016 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:58:46.054 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:58:46.104 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:58:46.159 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:58:46.333 INFO:teuthology.orchestra.run.smithi137.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.7 MB/s | 747 kB 00:00 2024-03-29T10:58:46.334 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:58:46.334 INFO:teuthology.orchestra.run.smithi137.stdout:Total 1.7 MB/s | 747 kB 00:00 2024-03-29T10:58:46.334 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T10:58:46.339 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T10:58:46.339 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T10:58:46.344 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T10:58:46.345 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:58:46.345 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T10:58:46.392 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:58:46.416 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T10:58:46.422 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:58:46.458 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:58:46.487 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:58:46.564 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:58:46.605 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:58:46.638 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:58:46.679 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:58:46.713 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:58:46.748 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:58:46.838 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:58:46.879 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:58:46.890 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:58:46.952 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:58:46.974 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:58:47.098 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:58:47.224 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:58:47.349 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:58:47.388 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:58:47.417 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:58:47.443 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:58:47.443 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:58:47.443 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:58:47.443 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:58:47.443 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:58:47.443 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:58:47.656 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:58:47.656 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:47.656 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T10:58:47.656 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:58:47.656 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:47.656 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T10:58:47.874 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:58:48.444 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:58 ago on Fri 29 Mar 2024 10:57:50 AM UTC. 2024-03-29T10:58:48.599 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T10:58:48.600 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:58:48.600 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-03-29T10:58:48.600 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:58:48.600 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T10:58:48.600 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:58:48.600 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:48.600 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T10:58:48.600 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:58:48.600 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-03-29T10:58:48.600 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:48.600 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 145 k 2024-03-29T10:58:48.601 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 436 k 2024-03-29T10:58:48.601 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T10:58:48.885 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:58:48.911 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:58:48.911 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:58:48.911 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:58:48.911 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:58:48.912 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:58:48.912 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:58:48.914 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 464 kB/s | 145 kB 00:00 2024-03-29T10:58:48.914 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:58:48.915 INFO:teuthology.orchestra.run.smithi137.stdout:Total 462 kB/s | 145 kB 00:00 2024-03-29T10:58:48.915 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T10:58:48.922 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T10:58:48.923 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T10:58:48.975 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T10:58:48.975 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T10:58:49.114 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T10:58:49.198 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:58:49.224 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:58:49.224 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:58:49.224 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:58:49.224 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:49.352 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:58:49.375 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:58:49.375 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:58:49.375 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:58:49.375 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:58:49.375 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:58:49.375 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:58:49.585 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:58:49.608 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:58:49.609 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:58:49.609 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:58:49.609 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:58:49.609 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:58:49.609 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:58:49.674 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:58:50.205 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:58:50.205 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:50.205 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T10:58:50.205 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:50.205 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:50.205 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T10:58:50.472 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr 2024-03-29T10:58:51.044 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:01 ago on Fri 29 Mar 2024 10:57:50 AM UTC. 2024-03-29T10:58:51.144 INFO:teuthology.orchestra.run.smithi137.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:58:51.196 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T10:58:51.197 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-03-29T10:58:51.197 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T10:58:51.268 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:58:51.447 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:58:51.447 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:58:51.447 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:58:51.447 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:58:51.448 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:58:51.448 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:58:51.448 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:58:51.448 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:58:51.448 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:58:51.448 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:58:51.448 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:58:51.448 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:58:51.448 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:58:51.448 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:58:51.448 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:58:51.448 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:58:51.448 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:58:51.448 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:58:51.449 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:58:51.449 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:58:51.449 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:58:51.449 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:58:51.449 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:58:51.449 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:58:51.449 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:58:51.449 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:58:51.449 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:58:51.449 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:58:51.449 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:58:51.449 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:58:51.449 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:58:51.449 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:58:51.450 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:58:51.450 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:58:51.450 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:58:51.450 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:58:51.829 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:01 ago on Fri 29 Mar 2024 10:57:50 AM UTC. 2024-03-29T10:58:51.978 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T10:58:51.979 INFO:teuthology.orchestra.run.smithi137.stdout:====================================================================================== 2024-03-29T10:58:51.979 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-29T10:58:51.980 INFO:teuthology.orchestra.run.smithi137.stdout:====================================================================================== 2024-03-29T10:58:51.980 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T10:58:51.980 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:58:51.980 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-29T10:58:51.980 INFO:teuthology.orchestra.run.smithi137.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:58:51.980 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:58:51.980 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:58:51.980 INFO:teuthology.orchestra.run.smithi137.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:58:51.980 INFO:teuthology.orchestra.run.smithi137.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:58:51.980 INFO:teuthology.orchestra.run.smithi137.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:58:51.980 INFO:teuthology.orchestra.run.smithi137.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:58:51.980 INFO:teuthology.orchestra.run.smithi137.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:58:51.980 INFO:teuthology.orchestra.run.smithi137.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:58:51.981 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:58:51.981 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:58:51.981 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:51.981 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T10:58:51.981 INFO:teuthology.orchestra.run.smithi137.stdout:====================================================================================== 2024-03-29T10:58:51.981 INFO:teuthology.orchestra.run.smithi137.stdout:Install 12 Packages 2024-03-29T10:58:51.981 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:51.982 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 6.9 M 2024-03-29T10:58:51.982 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 26 M 2024-03-29T10:58:51.982 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T10:58:52.277 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:58:52.278 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:58:52.278 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T10:58:52.278 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:52.278 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:52.278 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:52.278 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:58:52.278 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:52.278 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:52.278 INFO:teuthology.orchestra.run.smithi071.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:58:52.278 INFO:teuthology.orchestra.run.smithi071.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:58:52.278 INFO:teuthology.orchestra.run.smithi071.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:58:52.279 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:58:52.279 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:58:52.279 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:58:52.279 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:58:52.279 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:58:52.279 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:58:52.279 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:58:52.279 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:58:52.279 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:58:52.279 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:58:52.279 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:58:52.279 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:58:52.279 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:58:52.279 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:58:52.279 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:58:52.280 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:58:52.280 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:58:52.280 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:58:52.280 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:58:52.280 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:58:52.280 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:58:52.280 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:58:52.280 INFO:teuthology.orchestra.run.smithi071.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:58:52.280 INFO:teuthology.orchestra.run.smithi071.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:58:52.280 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:58:52.280 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:58:52.280 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:58:52.280 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:58:52.280 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T10:58:52.526 INFO:teuthology.orchestra.run.smithi137.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 82 kB/s | 15 kB 00:00 2024-03-29T10:58:52.584 INFO:teuthology.orchestra.run.smithi137.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 128 kB/s | 31 kB 00:00 2024-03-29T10:58:52.604 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-base 2024-03-29T10:58:52.685 INFO:teuthology.orchestra.run.smithi137.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-29T10:58:52.710 INFO:teuthology.orchestra.run.smithi137.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 238 kB 00:00 2024-03-29T10:58:52.860 INFO:teuthology.orchestra.run.smithi137.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.4 MB/s | 1.8 MB 00:00 2024-03-29T10:58:52.935 INFO:teuthology.orchestra.run.smithi137.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 259 kB/s | 19 kB 00:00 2024-03-29T10:58:52.961 INFO:teuthology.orchestra.run.smithi137.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.1 MB/s | 548 kB 00:00 2024-03-29T10:58:53.019 INFO:teuthology.orchestra.run.smithi137.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.5 MB/s | 863 kB 00:00 2024-03-29T10:58:53.044 INFO:teuthology.orchestra.run.smithi137.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.7 MB/s | 144 kB 00:00 2024-03-29T10:58:53.111 INFO:teuthology.orchestra.run.smithi137.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-29T10:58:53.137 INFO:teuthology.orchestra.run.smithi137.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 263 kB/s | 31 kB 00:00 2024-03-29T10:58:53.170 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:57 ago on Fri 29 Mar 2024 10:57:56 AM UTC. 2024-03-29T10:58:53.268 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:58:53.320 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T10:58:53.321 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-29T10:58:53.321 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T10:58:53.353 INFO:teuthology.orchestra.run.smithi137.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.9 MB/s | 2.0 MB 00:00 2024-03-29T10:58:53.356 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:58:53.356 INFO:teuthology.orchestra.run.smithi137.stdout:Total 5.0 MB/s | 6.9 MB 00:01 2024-03-29T10:58:53.374 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T10:58:53.391 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T10:58:53.391 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T10:58:53.403 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install cephadm 2024-03-29T10:58:53.563 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T10:58:53.563 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T10:58:53.965 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:57 ago on Fri 29 Mar 2024 10:57:56 AM UTC. 2024-03-29T10:58:54.070 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T10:58:54.115 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T10:58:54.116 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:58:54.116 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-29T10:58:54.116 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:58:54.116 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T10:58:54.116 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:58:54.116 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:58:54.116 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T10:58:54.117 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:58:54.117 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-29T10:58:54.117 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:58:54.117 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 747 k 2024-03-29T10:58:54.117 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 754 k 2024-03-29T10:58:54.117 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T10:58:54.145 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:58:54.189 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:58:54.236 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:58:54.266 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:58:54.373 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:58:54.499 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:58:54.594 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:58:54.637 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:58:54.649 INFO:teuthology.orchestra.run.smithi071.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.4 MB/s | 747 kB 00:00 2024-03-29T10:58:54.649 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:58:54.650 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.4 MB/s | 747 kB 00:00 2024-03-29T10:58:54.650 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T10:58:54.655 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T10:58:54.655 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T10:58:54.660 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T10:58:54.661 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T10:58:54.665 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:58:54.697 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:58:54.737 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T10:58:54.938 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:58:54.981 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:58:55.180 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:58:55.260 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:58:55.665 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:58:55.847 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:58:55.847 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:58:55.847 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:58:55.847 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:58:55.847 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:58:55.847 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:58:55.848 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:58:55.848 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:58:55.848 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:58:55.848 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:58:55.848 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:58:55.848 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:58:55.945 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:58:55.946 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:58:55.946 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T10:58:55.946 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:58:55.946 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:58:55.946 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T10:58:56.135 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:58:56.204 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:58:56.204 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:56.205 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T10:58:56.205 INFO:teuthology.orchestra.run.smithi137.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:58:56.205 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:58:56.205 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:58:56.205 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:58:56.205 INFO:teuthology.orchestra.run.smithi137.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:58:56.205 INFO:teuthology.orchestra.run.smithi137.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:58:56.205 INFO:teuthology.orchestra.run.smithi137.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:58:56.205 INFO:teuthology.orchestra.run.smithi137.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:58:56.205 INFO:teuthology.orchestra.run.smithi137.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:58:56.205 INFO:teuthology.orchestra.run.smithi137.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:58:56.205 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:58:56.206 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:58:56.206 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:56.206 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T10:58:56.450 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:58:56.702 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 10:57:56 AM UTC. 2024-03-29T10:58:56.854 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T10:58:56.855 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:58:56.855 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-29T10:58:56.855 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:58:56.855 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T10:58:56.855 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:58:56.855 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:58:56.856 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T10:58:56.856 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:58:56.856 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-29T10:58:56.856 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:58:56.856 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 145 k 2024-03-29T10:58:56.856 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 436 k 2024-03-29T10:58:56.856 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T10:58:57.010 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:07 ago on Fri 29 Mar 2024 10:57:50 AM UTC. 2024-03-29T10:58:57.079 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 653 kB/s | 145 kB 00:00 2024-03-29T10:58:57.079 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:58:57.080 INFO:teuthology.orchestra.run.smithi071.stdout:Total 647 kB/s | 145 kB 00:00 2024-03-29T10:58:57.080 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T10:58:57.087 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T10:58:57.087 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T10:58:57.139 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T10:58:57.139 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T10:58:57.162 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T10:58:57.164 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================================ 2024-03-29T10:58:57.164 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-29T10:58:57.164 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================================ 2024-03-29T10:58:57.164 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T10:58:57.164 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:58:57.164 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-29T10:58:57.164 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:58:57.164 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:58:57.164 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:58:57.164 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:58:57.164 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:58:57.164 INFO:teuthology.orchestra.run.smithi137.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:58:57.165 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:58:57.165 INFO:teuthology.orchestra.run.smithi137.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:58:57.165 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:58:57.165 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:58:57.165 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:58:57.165 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:57.165 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T10:58:57.165 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================================ 2024-03-29T10:58:57.165 INFO:teuthology.orchestra.run.smithi137.stdout:Install 12 Packages 2024-03-29T10:58:57.165 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:58:57.166 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 37 M 2024-03-29T10:58:57.166 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 209 M 2024-03-29T10:58:57.166 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T10:58:57.278 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T10:58:57.347 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:58:57.371 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:58:57.371 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:58:57.371 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:58:57.371 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:58:57.702 INFO:teuthology.orchestra.run.smithi137.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 225 kB/s | 32 kB 00:00 2024-03-29T10:58:57.735 INFO:teuthology.orchestra.run.smithi137.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 541 kB/s | 17 kB 00:00 2024-03-29T10:58:57.794 INFO:teuthology.orchestra.run.smithi137.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 592 kB/s | 34 kB 00:00 2024-03-29T10:58:57.844 INFO:teuthology.orchestra.run.smithi137.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-29T10:58:58.053 INFO:teuthology.orchestra.run.smithi137.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-03-29T10:58:58.146 INFO:teuthology.orchestra.run.smithi137.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 4.8 MB 00:00 2024-03-29T10:58:58.178 INFO:teuthology.orchestra.run.smithi137.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.7 MB/s | 217 kB 00:00 2024-03-29T10:58:58.379 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:58:58.379 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:58:58.379 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T10:58:58.379 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:58:58.380 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:58:58.380 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T10:58:58.512 INFO:teuthology.orchestra.run.smithi137.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-03-29T10:58:58.603 INFO:teuthology.orchestra.run.smithi137.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 803 kB 00:00 2024-03-29T10:58:58.630 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr 2024-03-29T10:58:58.704 INFO:teuthology.orchestra.run.smithi137.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-03-29T10:58:58.804 INFO:teuthology.orchestra.run.smithi137.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-1 5.9 MB/s | 7.4 MB 00:01 2024-03-29T10:58:58.829 INFO:teuthology.orchestra.run.smithi137.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-03-29T10:58:58.831 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:58:58.831 INFO:teuthology.orchestra.run.smithi137.stdout:Total 22 MB/s | 37 MB 00:01 2024-03-29T10:58:58.873 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T10:58:58.897 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T10:58:58.898 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T10:58:59.207 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:03 ago on Fri 29 Mar 2024 10:57:56 AM UTC. 2024-03-29T10:58:59.258 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T10:58:59.259 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T10:58:59.306 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:58:59.358 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T10:58:59.359 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-29T10:58:59.359 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T10:58:59.429 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:58:59.968 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T10:58:59.990 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:04 ago on Fri 29 Mar 2024 10:57:56 AM UTC. 2024-03-29T10:59:00.052 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:59:00.083 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:59:00.141 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T10:59:00.142 INFO:teuthology.orchestra.run.smithi071.stdout:====================================================================================== 2024-03-29T10:59:00.142 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-29T10:59:00.142 INFO:teuthology.orchestra.run.smithi071.stdout:====================================================================================== 2024-03-29T10:59:00.143 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T10:59:00.143 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:59:00.143 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-29T10:59:00.143 INFO:teuthology.orchestra.run.smithi071.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:59:00.143 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:59:00.143 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:59:00.143 INFO:teuthology.orchestra.run.smithi071.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:59:00.143 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:59:00.143 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:59:00.143 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:59:00.143 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:59:00.143 INFO:teuthology.orchestra.run.smithi071.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:59:00.143 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:59:00.143 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:59:00.144 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:00.144 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T10:59:00.144 INFO:teuthology.orchestra.run.smithi071.stdout:====================================================================================== 2024-03-29T10:59:00.144 INFO:teuthology.orchestra.run.smithi071.stdout:Install 12 Packages 2024-03-29T10:59:00.144 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:00.144 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 6.9 M 2024-03-29T10:59:00.144 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 26 M 2024-03-29T10:59:00.145 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T10:59:00.176 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:59:00.212 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:59:00.390 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:59:00.419 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:59:00.495 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:59:00.690 INFO:teuthology.orchestra.run.smithi071.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 165 kB/s | 31 kB 00:00 2024-03-29T10:59:00.799 INFO:teuthology.orchestra.run.smithi071.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 51 kB/s | 15 kB 00:00 2024-03-29T10:59:00.857 INFO:teuthology.orchestra.run.smithi071.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.0 MB/s | 1.0 MB 00:00 2024-03-29T10:59:00.883 INFO:teuthology.orchestra.run.smithi071.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-03-29T10:59:01.083 INFO:teuthology.orchestra.run.smithi071.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.7 MB/s | 548 kB 00:00 2024-03-29T10:59:01.125 INFO:teuthology.orchestra.run.smithi071.stdout:(6/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 2.9 MB/s | 1.8 MB 00:00 2024-03-29T10:59:01.142 INFO:teuthology.orchestra.run.smithi071.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 331 kB/s | 19 kB 00:00 2024-03-29T10:59:01.150 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:59:01.200 INFO:teuthology.orchestra.run.smithi071.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.4 MB/s | 144 kB 00:00 2024-03-29T10:59:01.234 INFO:teuthology.orchestra.run.smithi071.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 924 kB/s | 31 kB 00:00 2024-03-29T10:59:01.272 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:59:01.293 INFO:teuthology.orchestra.run.smithi071.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-03-29T10:59:01.584 INFO:teuthology.orchestra.run.smithi071.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.4 MB/s | 2.0 MB 00:00 2024-03-29T10:59:01.668 INFO:teuthology.orchestra.run.smithi071.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.0 MB/s | 863 kB 00:00 2024-03-29T10:59:01.671 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:59:01.671 INFO:teuthology.orchestra.run.smithi071.stdout:Total 4.5 MB/s | 6.9 MB 00:01 2024-03-29T10:59:01.689 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T10:59:01.706 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T10:59:01.706 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T10:59:01.880 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T10:59:01.881 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T10:59:02.391 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T10:59:02.476 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:59:02.512 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:59:02.519 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:59:02.567 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:59:02.597 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:59:02.708 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:59:02.831 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:59:02.933 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:59:02.985 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:59:03.013 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:59:03.045 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:59:03.287 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:59:03.312 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:59:03.545 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:59:03.576 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:59:04.159 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:59:04.159 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:59:04.159 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:59:04.159 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:59:04.159 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:59:04.160 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:59:04.160 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:59:04.160 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:59:04.160 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:59:04.160 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:59:04.160 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:59:04.160 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:59:04.501 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:59:04.502 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:04.502 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T10:59:04.502 INFO:teuthology.orchestra.run.smithi071.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:59:04.502 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:59:04.502 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:59:04.502 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:59:04.502 INFO:teuthology.orchestra.run.smithi071.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:59:04.502 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:59:04.502 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:59:04.502 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:59:04.502 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:59:04.502 INFO:teuthology.orchestra.run.smithi071.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:59:04.502 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:59:04.503 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:59:04.503 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:04.503 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T10:59:04.747 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:59:05.300 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:09 ago on Fri 29 Mar 2024 10:57:56 AM UTC. 2024-03-29T10:59:05.452 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T10:59:05.453 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================================ 2024-03-29T10:59:05.453 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-29T10:59:05.453 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================================ 2024-03-29T10:59:05.453 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T10:59:05.453 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:59:05.453 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-29T10:59:05.453 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:59:05.454 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:59:05.454 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:59:05.454 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:59:05.454 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:59:05.454 INFO:teuthology.orchestra.run.smithi071.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:59:05.454 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:59:05.454 INFO:teuthology.orchestra.run.smithi071.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:59:05.454 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:59:05.454 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:59:05.454 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:59:05.454 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:05.454 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T10:59:05.455 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================================ 2024-03-29T10:59:05.455 INFO:teuthology.orchestra.run.smithi071.stdout:Install 12 Packages 2024-03-29T10:59:05.455 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:05.455 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 37 M 2024-03-29T10:59:05.455 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 209 M 2024-03-29T10:59:05.455 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T10:59:05.716 INFO:teuthology.orchestra.run.smithi071.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 290 kB/s | 32 kB 00:00 2024-03-29T10:59:05.774 INFO:teuthology.orchestra.run.smithi071.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 486 kB/s | 17 kB 00:00 2024-03-29T10:59:05.808 INFO:teuthology.orchestra.run.smithi071.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-29T10:59:05.983 INFO:teuthology.orchestra.run.smithi071.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2024-03-29T10:59:06.142 INFO:teuthology.orchestra.run.smithi071.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 4.8 MB 00:00 2024-03-29T10:59:06.161 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:59:06.161 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:59:06.161 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:59:06.162 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:59:06.162 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:59:06.162 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:59:06.162 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:59:06.162 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:59:06.162 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:59:06.162 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:59:06.162 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:59:06.162 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:59:06.183 INFO:teuthology.orchestra.run.smithi071.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.2 MB/s | 217 kB 00:00 2024-03-29T10:59:06.301 INFO:teuthology.orchestra.run.smithi071.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.3 MB/s | 3.0 MB 00:00 2024-03-29T10:59:06.605 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:59:06.605 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:06.605 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T10:59:06.605 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:59:06.605 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:59:06.605 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:59:06.605 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:59:06.605 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:59:06.605 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:59:06.606 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:59:06.606 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:59:06.606 INFO:teuthology.orchestra.run.smithi137.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:59:06.606 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:59:06.606 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:59:06.606 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:59:06.606 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:06.607 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T10:59:06.652 INFO:teuthology.orchestra.run.smithi071.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-03-29T10:59:06.812 INFO:teuthology.orchestra.run.smithi071.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-03-29T10:59:06.964 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:59:07.118 INFO:teuthology.orchestra.run.smithi071.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 986 kB/s | 803 kB 00:00 2024-03-29T10:59:07.343 INFO:teuthology.orchestra.run.smithi071.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 460 kB/s | 245 kB 00:00 2024-03-29T10:59:07.536 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:17 ago on Fri 29 Mar 2024 10:57:50 AM UTC. 2024-03-29T10:59:07.688 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T10:59:07.689 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================== 2024-03-29T10:59:07.689 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-29T10:59:07.689 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================== 2024-03-29T10:59:07.689 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T10:59:07.689 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:59:07.689 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-29T10:59:07.689 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:59:07.689 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:59:07.689 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:59:07.689 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:59:07.689 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:59:07.690 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:59:07.690 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:59:07.690 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:59:07.690 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:59:07.690 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:59:07.690 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:59:07.690 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:59:07.690 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:07.690 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T10:59:07.690 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================== 2024-03-29T10:59:07.690 INFO:teuthology.orchestra.run.smithi137.stdout:Install 13 Packages 2024-03-29T10:59:07.690 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:07.691 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 2.0 M 2024-03-29T10:59:07.691 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 26 M 2024-03-29T10:59:07.691 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T10:59:07.893 INFO:teuthology.orchestra.run.smithi137.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 158 kB/s | 24 kB 00:00 2024-03-29T10:59:07.910 INFO:teuthology.orchestra.run.smithi137.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 105 kB/s | 18 kB 00:00 2024-03-29T10:59:07.977 INFO:teuthology.orchestra.run.smithi137.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-29T10:59:08.002 INFO:teuthology.orchestra.run.smithi137.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-03-29T10:59:08.028 INFO:teuthology.orchestra.run.smithi137.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T10:59:08.044 INFO:teuthology.orchestra.run.smithi137.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-29T10:59:08.061 INFO:teuthology.orchestra.run.smithi137.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 162 kB/s | 52 kB 00:00 2024-03-29T10:59:08.195 INFO:teuthology.orchestra.run.smithi137.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 100 kB/s | 15 kB 00:00 2024-03-29T10:59:08.212 INFO:teuthology.orchestra.run.smithi137.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 175 kB/s | 32 kB 00:00 2024-03-29T10:59:08.354 INFO:teuthology.orchestra.run.smithi137.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 417 kB/s | 59 kB 00:00 2024-03-29T10:59:08.379 INFO:teuthology.orchestra.run.smithi137.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 639 kB/s | 203 kB 00:00 2024-03-29T10:59:08.454 INFO:teuthology.orchestra.run.smithi137.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 892 kB/s | 90 kB 00:00 2024-03-29T10:59:08.496 INFO:teuthology.orchestra.run.smithi137.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.4 MB/s | 1.0 MB 00:00 2024-03-29T10:59:08.498 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:59:08.498 INFO:teuthology.orchestra.run.smithi137.stdout:Total 2.5 MB/s | 2.0 MB 00:00 2024-03-29T10:59:08.498 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T10:59:08.511 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T10:59:08.511 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T10:59:08.600 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T10:59:08.601 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T10:59:09.155 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T10:59:09.276 INFO:teuthology.orchestra.run.smithi071.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 2.0 MB/s | 7.4 MB 00:03 2024-03-29T10:59:09.279 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:59:09.279 INFO:teuthology.orchestra.run.smithi071.stdout:Total 9.7 MB/s | 37 MB 00:03 2024-03-29T10:59:09.322 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T10:59:09.332 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:59:09.349 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T10:59:09.349 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T10:59:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:59:09.444 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:59:09.482 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:59:09.522 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:59:09.616 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:59:09.711 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T10:59:09.711 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T10:59:09.716 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:59:09.774 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:59:10.434 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T10:59:10.496 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:59:10.499 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:59:10.538 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:59:10.539 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:59:10.572 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:59:10.620 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:59:10.632 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:59:10.642 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:59:10.667 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:59:10.843 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:59:10.874 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:59:10.950 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:59:11.474 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:59:11.474 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:59:11.474 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:59:11.475 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:59:11.475 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:59:11.475 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:59:11.475 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:59:11.475 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:59:11.475 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:59:11.475 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:59:11.475 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:59:11.475 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:59:11.475 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:59:11.602 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:59:11.703 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:59:11.835 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:59:11.835 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:11.835 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T10:59:11.836 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:59:11.836 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:59:11.836 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:59:11.836 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:59:11.836 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:59:11.836 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:59:11.836 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:59:11.836 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:59:11.836 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:59:11.836 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:59:11.836 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:59:11.836 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:59:11.836 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:59:11.837 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:11.837 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T10:59:12.120 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:59:12.693 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:22 ago on Fri 29 Mar 2024 10:57:50 AM UTC. 2024-03-29T10:59:12.848 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T10:59:12.848 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================== 2024-03-29T10:59:12.848 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-29T10:59:12.848 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================== 2024-03-29T10:59:12.849 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T10:59:12.849 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:59:12.849 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-29T10:59:12.849 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:59:12.849 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:59:12.849 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:59:12.849 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:59:12.849 INFO:teuthology.orchestra.run.smithi137.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:59:12.849 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:12.849 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T10:59:12.849 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================== 2024-03-29T10:59:12.849 INFO:teuthology.orchestra.run.smithi137.stdout:Install 6 Packages 2024-03-29T10:59:12.849 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:12.850 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 6.8 M 2024-03-29T10:59:12.850 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 33 M 2024-03-29T10:59:12.850 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T10:59:12.935 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:59:13.314 INFO:teuthology.orchestra.run.smithi137.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-29T10:59:13.340 INFO:teuthology.orchestra.run.smithi137.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 477 kB/s | 138 kB 00:00 2024-03-29T10:59:13.373 INFO:teuthology.orchestra.run.smithi137.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 769 kB/s | 248 kB 00:00 2024-03-29T10:59:13.673 INFO:teuthology.orchestra.run.smithi137.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 173 kB/s | 58 kB 00:00 2024-03-29T10:59:13.698 INFO:teuthology.orchestra.run.smithi137.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 157 kB/s | 51 kB 00:00 2024-03-29T10:59:13.941 INFO:teuthology.orchestra.run.smithi137.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 876 kB/s | 548 kB 00:00 2024-03-29T10:59:13.942 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:59:13.942 INFO:teuthology.orchestra.run.smithi137.stdout:Total 6.2 MB/s | 6.8 MB 00:01 2024-03-29T10:59:13.952 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:59:13.956 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T10:59:13.966 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T10:59:13.966 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T10:59:13.981 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:59:14.054 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T10:59:14.055 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T10:59:14.351 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T10:59:14.504 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:59:14.550 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:59:15.060 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:59:15.154 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:59:15.214 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:59:15.238 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:59:16.062 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:59:16.062 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:59:16.062 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:59:16.062 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:59:16.062 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:59:16.062 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:59:16.367 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:59:16.367 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:16.367 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T10:59:16.367 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:59:16.367 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:59:16.367 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:59:16.367 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:59:16.367 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:59:16.368 INFO:teuthology.orchestra.run.smithi137.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:59:16.368 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:16.368 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T10:59:16.584 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:59:16.584 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:59:16.584 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:59:16.584 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:59:16.585 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:59:16.585 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:59:16.585 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:59:16.585 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:59:16.585 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:59:16.585 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:59:16.585 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:59:16.585 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:59:16.624 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-fuse 2024-03-29T10:59:17.018 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:59:17.019 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:17.019 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T10:59:17.019 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:59:17.019 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:59:17.019 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:59:17.019 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:59:17.019 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:59:17.019 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:59:17.019 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:59:17.020 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:59:17.020 INFO:teuthology.orchestra.run.smithi071.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:59:17.020 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:59:17.020 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:59:17.020 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:59:17.020 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:17.020 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T10:59:17.185 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:27 ago on Fri 29 Mar 2024 10:57:50 AM UTC. 2024-03-29T10:59:17.335 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T10:59:17.336 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:59:17.336 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-29T10:59:17.336 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:59:17.336 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T10:59:17.336 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:59:17.336 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-29T10:59:17.336 INFO:teuthology.orchestra.run.smithi137.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:59:17.336 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:17.336 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T10:59:17.336 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:59:17.336 INFO:teuthology.orchestra.run.smithi137.stdout:Install 2 Packages 2024-03-29T10:59:17.336 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:17.337 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 964 k 2024-03-29T10:59:17.337 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 2.8 M 2024-03-29T10:59:17.337 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T10:59:17.369 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:59:17.467 INFO:teuthology.orchestra.run.smithi137.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 607 kB/s | 79 kB 00:00 2024-03-29T10:59:17.800 INFO:teuthology.orchestra.run.smithi137.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.9 MB/s | 885 kB 00:00 2024-03-29T10:59:17.801 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:59:17.802 INFO:teuthology.orchestra.run.smithi137.stdout:Total 2.0 MB/s | 964 kB 00:00 2024-03-29T10:59:17.802 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T10:59:17.822 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T10:59:17.822 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T10:59:17.896 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T10:59:17.896 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T10:59:17.939 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:21 ago on Fri 29 Mar 2024 10:57:56 AM UTC. 2024-03-29T10:59:18.077 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T10:59:18.097 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T10:59:18.098 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2024-03-29T10:59:18.098 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-29T10:59:18.098 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2024-03-29T10:59:18.099 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T10:59:18.099 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:59:18.099 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-29T10:59:18.099 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:59:18.099 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:59:18.099 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:59:18.099 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:59:18.099 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:59:18.099 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:59:18.099 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:59:18.099 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:59:18.099 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:59:18.099 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:59:18.100 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:59:18.100 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:59:18.100 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:18.100 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T10:59:18.100 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2024-03-29T10:59:18.100 INFO:teuthology.orchestra.run.smithi071.stdout:Install 13 Packages 2024-03-29T10:59:18.100 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:18.101 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 2.0 M 2024-03-29T10:59:18.101 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 26 M 2024-03-29T10:59:18.101 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T10:59:18.218 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:59:18.292 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:59:18.581 INFO:teuthology.orchestra.run.smithi071.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 193 kB/s | 24 kB 00:00 2024-03-29T10:59:18.606 INFO:teuthology.orchestra.run.smithi071.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 119 kB/s | 18 kB 00:00 2024-03-29T10:59:18.656 INFO:teuthology.orchestra.run.smithi071.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T10:59:18.681 INFO:teuthology.orchestra.run.smithi071.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T10:59:18.707 INFO:teuthology.orchestra.run.smithi071.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T10:59:18.732 INFO:teuthology.orchestra.run.smithi071.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 858 kB/s | 43 kB 00:00 2024-03-29T10:59:18.757 INFO:teuthology.orchestra.run.smithi071.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 172 kB/s | 52 kB 00:00 2024-03-29T10:59:18.960 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:59:18.960 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:59:18.974 INFO:teuthology.orchestra.run.smithi071.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 62 kB/s | 15 kB 00:00 2024-03-29T10:59:19.024 INFO:teuthology.orchestra.run.smithi071.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 101 kB/s | 32 kB 00:00 2024-03-29T10:59:19.166 INFO:teuthology.orchestra.run.smithi071.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 417 kB/s | 59 kB 00:00 2024-03-29T10:59:19.191 INFO:teuthology.orchestra.run.smithi071.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 467 kB/s | 203 kB 00:00 2024-03-29T10:59:19.266 INFO:teuthology.orchestra.run.smithi071.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 897 kB/s | 90 kB 00:00 2024-03-29T10:59:19.274 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:59:19.274 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:19.274 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T10:59:19.274 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:59:19.274 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:19.274 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T10:59:19.350 INFO:teuthology.orchestra.run.smithi071.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.7 MB/s | 1.0 MB 00:00 2024-03-29T10:59:19.354 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:59:19.354 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-03-29T10:59:19.354 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T10:59:19.368 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T10:59:19.368 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T10:59:19.459 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T10:59:19.460 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T10:59:19.552 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-volume 2024-03-29T10:59:19.967 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T10:59:20.129 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:30 ago on Fri 29 Mar 2024 10:57:50 AM UTC. 2024-03-29T10:59:20.146 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:59:20.192 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:59:20.248 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:59:20.285 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T10:59:20.286 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:59:20.286 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:59:20.286 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-29T10:59:20.286 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:59:20.286 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T10:59:20.286 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:59:20.286 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-29T10:59:20.286 INFO:teuthology.orchestra.run.smithi137.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:59:20.286 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:20.287 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T10:59:20.287 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:59:20.287 INFO:teuthology.orchestra.run.smithi137.stdout:Install 2 Packages 2024-03-29T10:59:20.287 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:20.287 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 332 k 2024-03-29T10:59:20.287 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 1.5 M 2024-03-29T10:59:20.288 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T10:59:20.327 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:59:20.413 INFO:teuthology.orchestra.run.smithi137.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 549 kB/s | 69 kB 00:00 2024-03-29T10:59:20.420 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:59:20.529 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:59:20.586 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:59:20.655 INFO:teuthology.orchestra.run.smithi137.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 718 kB/s | 264 kB 00:00 2024-03-29T10:59:20.656 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:59:20.656 INFO:teuthology.orchestra.run.smithi137.stdout:Total 900 kB/s | 332 kB 00:00 2024-03-29T10:59:20.656 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T10:59:20.665 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T10:59:20.665 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T10:59:20.702 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T10:59:20.702 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T10:59:20.855 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T10:59:21.039 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:59:21.074 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:59:21.090 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:59:21.090 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:59:21.090 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:59:21.090 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:21.308 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:59:21.358 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:59:21.392 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:59:21.440 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:59:21.462 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:59:21.814 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:59:22.131 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:59:22.131 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:22.131 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T10:59:22.131 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:59:22.131 INFO:teuthology.orchestra.run.smithi137.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:59:22.131 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:22.131 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T10:59:22.289 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:59:22.290 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:59:22.290 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:59:22.291 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:59:22.292 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:59:22.292 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:59:22.292 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:59:22.292 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:59:22.292 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:59:22.292 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:59:22.292 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:59:22.292 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:59:22.292 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:59:22.546 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librados-devel 2024-03-29T10:59:22.648 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:59:22.648 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:22.648 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T10:59:22.648 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:59:22.649 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:59:22.649 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:59:22.649 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:59:22.649 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:59:22.649 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:59:22.649 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:59:22.649 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:59:22.649 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:59:22.649 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:59:22.649 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:59:22.649 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:59:22.649 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:59:22.649 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:22.649 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T10:59:22.896 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:59:23.125 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Mar 2024 10:57:50 AM UTC. 2024-03-29T10:59:23.277 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T10:59:23.278 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:59:23.278 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-03-29T10:59:23.278 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:59:23.278 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T10:59:23.278 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:59:23.278 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:23.278 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T10:59:23.278 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:59:23.278 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-03-29T10:59:23.278 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:23.279 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 127 k 2024-03-29T10:59:23.279 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 456 k 2024-03-29T10:59:23.279 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T10:59:23.461 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:27 ago on Fri 29 Mar 2024 10:57:56 AM UTC. 2024-03-29T10:59:23.477 INFO:teuthology.orchestra.run.smithi137.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 640 kB/s | 127 kB 00:00 2024-03-29T10:59:23.477 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:59:23.478 INFO:teuthology.orchestra.run.smithi137.stdout:Total 635 kB/s | 127 kB 00:00 2024-03-29T10:59:23.478 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T10:59:23.483 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T10:59:23.483 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T10:59:23.538 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T10:59:23.539 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T10:59:23.613 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T10:59:23.613 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2024-03-29T10:59:23.614 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-29T10:59:23.614 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2024-03-29T10:59:23.614 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T10:59:23.614 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:59:23.614 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-29T10:59:23.614 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:59:23.614 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:59:23.614 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:59:23.614 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:59:23.614 INFO:teuthology.orchestra.run.smithi071.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:59:23.614 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:23.615 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T10:59:23.615 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2024-03-29T10:59:23.615 INFO:teuthology.orchestra.run.smithi071.stdout:Install 6 Packages 2024-03-29T10:59:23.615 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:23.615 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 6.8 M 2024-03-29T10:59:23.615 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 33 M 2024-03-29T10:59:23.615 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T10:59:23.690 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T10:59:23.783 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:24.201 INFO:teuthology.orchestra.run.smithi071.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-29T10:59:24.226 INFO:teuthology.orchestra.run.smithi071.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 488 kB/s | 138 kB 00:00 2024-03-29T10:59:24.231 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:24.294 INFO:teuthology.orchestra.run.smithi071.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 709 kB/s | 248 kB 00:00 2024-03-29T10:59:24.352 INFO:teuthology.orchestra.run.smithi071.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 461 kB/s | 58 kB 00:00 2024-03-29T10:59:24.410 INFO:teuthology.orchestra.run.smithi071.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 437 kB/s | 51 kB 00:00 2024-03-29T10:59:24.516 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:24.516 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:24.516 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T10:59:24.516 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:59:24.516 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:24.517 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T10:59:24.594 INFO:teuthology.orchestra.run.smithi071.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.4 MB/s | 548 kB 00:00 2024-03-29T10:59:24.596 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:59:24.596 INFO:teuthology.orchestra.run.smithi071.stdout:Total 6.9 MB/s | 6.8 MB 00:00 2024-03-29T10:59:24.609 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T10:59:24.620 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T10:59:24.620 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T10:59:24.710 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T10:59:24.710 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T10:59:24.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install libcephfs2 2024-03-29T10:59:25.005 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T10:59:25.166 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:59:25.220 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:59:25.344 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 10:57:50 AM UTC. 2024-03-29T10:59:25.449 INFO:teuthology.orchestra.run.smithi137.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:59:25.502 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T10:59:25.503 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-03-29T10:59:25.503 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T10:59:25.580 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install libcephfs-devel 2024-03-29T10:59:25.744 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:59:25.822 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:59:25.884 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:59:25.935 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:59:26.152 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 10:57:50 AM UTC. 2024-03-29T10:59:26.304 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T10:59:26.305 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:59:26.305 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-03-29T10:59:26.305 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:59:26.305 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T10:59:26.305 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:59:26.305 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:26.305 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T10:59:26.305 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:59:26.305 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-03-29T10:59:26.306 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:26.306 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 31 k 2024-03-29T10:59:26.306 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 140 k 2024-03-29T10:59:26.306 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T10:59:26.473 INFO:teuthology.orchestra.run.smithi137.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 186 kB/s | 31 kB 00:00 2024-03-29T10:59:26.474 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:59:26.474 INFO:teuthology.orchestra.run.smithi137.stdout:Total 185 kB/s | 31 kB 00:00 2024-03-29T10:59:26.474 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T10:59:26.477 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T10:59:26.477 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T10:59:26.498 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T10:59:26.498 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T10:59:26.639 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T10:59:26.695 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:59:26.695 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:59:26.695 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:59:26.695 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:59:26.695 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:59:26.695 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:59:26.756 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:26.997 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:59:26.997 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:26.997 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T10:59:26.997 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:59:26.997 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:59:26.997 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:59:26.997 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:59:26.997 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:59:26.997 INFO:teuthology.orchestra.run.smithi071.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:59:26.998 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:26.998 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T10:59:27.134 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:27.239 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-fuse 2024-03-29T10:59:27.406 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:27.426 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:27.427 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T10:59:27.427 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:59:27.427 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:27.427 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T10:59:27.583 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librados2 2024-03-29T10:59:27.806 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:31 ago on Fri 29 Mar 2024 10:57:56 AM UTC. 2024-03-29T10:59:27.954 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T10:59:27.955 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:59:27.955 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-29T10:59:27.955 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:59:27.955 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T10:59:27.955 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:59:27.955 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-29T10:59:27.955 INFO:teuthology.orchestra.run.smithi071.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:59:27.955 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:27.955 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T10:59:27.956 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:59:27.956 INFO:teuthology.orchestra.run.smithi071.stdout:Install 2 Packages 2024-03-29T10:59:27.956 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:27.956 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 964 k 2024-03-29T10:59:27.956 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 2.8 M 2024-03-29T10:59:27.956 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T10:59:28.086 INFO:teuthology.orchestra.run.smithi071.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 605 kB/s | 79 kB 00:00 2024-03-29T10:59:28.152 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 10:57:50 AM UTC. 2024-03-29T10:59:28.250 INFO:teuthology.orchestra.run.smithi137.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:59:28.287 INFO:teuthology.orchestra.run.smithi071.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.6 MB/s | 885 kB 00:00 2024-03-29T10:59:28.288 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:59:28.288 INFO:teuthology.orchestra.run.smithi071.stdout:Total 2.8 MB/s | 964 kB 00:00 2024-03-29T10:59:28.288 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T10:59:28.301 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T10:59:28.302 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-03-29T10:59:28.302 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T10:59:28.308 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T10:59:28.308 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T10:59:28.359 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librbd1 2024-03-29T10:59:28.381 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T10:59:28.382 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T10:59:28.571 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T10:59:28.712 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:59:28.779 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:59:28.922 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 10:57:50 AM UTC. 2024-03-29T10:59:29.020 INFO:teuthology.orchestra.run.smithi137.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:59:29.071 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T10:59:29.072 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-03-29T10:59:29.072 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T10:59:29.122 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rados 2024-03-29T10:59:29.419 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:59:29.419 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:59:29.690 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 10:57:50 AM UTC. 2024-03-29T10:59:29.737 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:59:29.737 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:29.737 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T10:59:29.738 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:59:29.738 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:29.738 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T10:59:29.789 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:59:29.840 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T10:59:29.842 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-03-29T10:59:29.842 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T10:59:29.892 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rgw 2024-03-29T10:59:30.059 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-volume 2024-03-29T10:59:30.454 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 10:57:50 AM UTC. 2024-03-29T10:59:30.553 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:59:30.605 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T10:59:30.606 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-03-29T10:59:30.606 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T10:59:30.628 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:34 ago on Fri 29 Mar 2024 10:57:56 AM UTC. 2024-03-29T10:59:30.663 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-cephfs 2024-03-29T10:59:30.782 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T10:59:30.782 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:59:30.783 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-29T10:59:30.783 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:59:30.783 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T10:59:30.783 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:59:30.783 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-29T10:59:30.783 INFO:teuthology.orchestra.run.smithi071.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:59:30.783 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:30.783 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T10:59:30.783 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:59:30.783 INFO:teuthology.orchestra.run.smithi071.stdout:Install 2 Packages 2024-03-29T10:59:30.783 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:30.784 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 332 k 2024-03-29T10:59:30.784 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 1.5 M 2024-03-29T10:59:30.784 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T10:59:30.899 INFO:teuthology.orchestra.run.smithi071.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 600 kB/s | 69 kB 00:00 2024-03-29T10:59:31.229 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:57:50 AM UTC. 2024-03-29T10:59:31.274 INFO:teuthology.orchestra.run.smithi071.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 538 kB/s | 264 kB 00:00 2024-03-29T10:59:31.275 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:59:31.275 INFO:teuthology.orchestra.run.smithi071.stdout:Total 675 kB/s | 332 kB 00:00 2024-03-29T10:59:31.276 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T10:59:31.284 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T10:59:31.284 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T10:59:31.320 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T10:59:31.320 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T10:59:31.327 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:59:31.379 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T10:59:31.380 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-03-29T10:59:31.380 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T10:59:31.435 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rbd 2024-03-29T10:59:31.473 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T10:59:31.655 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:59:31.693 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:59:31.709 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:59:31.709 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:59:31.709 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:59:31.709 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:31.999 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:57:50 AM UTC. 2024-03-29T10:59:32.098 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:59:32.149 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T10:59:32.150 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-03-29T10:59:32.150 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T10:59:32.209 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-fuse 2024-03-29T10:59:32.436 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:59:32.750 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:59:32.751 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:32.751 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T10:59:32.751 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:59:32.751 INFO:teuthology.orchestra.run.smithi071.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:59:32.751 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:32.751 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T10:59:32.773 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:57:50 AM UTC. 2024-03-29T10:59:32.923 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T10:59:32.924 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:59:32.924 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-03-29T10:59:32.924 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:59:32.924 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T10:59:32.924 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:59:32.924 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:32.924 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T10:59:32.924 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:59:32.925 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-03-29T10:59:32.925 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:32.925 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 85 k 2024-03-29T10:59:32.925 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 231 k 2024-03-29T10:59:32.925 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T10:59:33.113 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados-devel 2024-03-29T10:59:33.384 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 186 kB/s | 85 kB 00:00 2024-03-29T10:59:33.384 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:59:33.384 INFO:teuthology.orchestra.run.smithi137.stdout:Total 185 kB/s | 85 kB 00:00 2024-03-29T10:59:33.385 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T10:59:33.392 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T10:59:33.392 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T10:59:33.448 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T10:59:33.449 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T10:59:33.590 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T10:59:33.682 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:33.688 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 10:57:56 AM UTC. 2024-03-29T10:59:33.839 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T10:59:33.840 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:59:33.840 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-29T10:59:33.840 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:59:33.841 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T10:59:33.841 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:59:33.841 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:33.841 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T10:59:33.841 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:59:33.841 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-29T10:59:33.841 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:33.841 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 127 k 2024-03-29T10:59:33.841 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 456 k 2024-03-29T10:59:33.841 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T10:59:34.116 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:34.434 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:34.435 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:34.435 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T10:59:34.435 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:59:34.435 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:34.435 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T10:59:34.462 INFO:teuthology.orchestra.run.smithi071.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 204 kB/s | 127 kB 00:00 2024-03-29T10:59:34.462 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:59:34.463 INFO:teuthology.orchestra.run.smithi071.stdout:Total 204 kB/s | 127 kB 00:00 2024-03-29T10:59:34.463 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T10:59:34.468 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T10:59:34.468 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T10:59:34.524 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T10:59:34.525 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T10:59:34.667 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T10:59:34.705 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-mirror 2024-03-29T10:59:34.769 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:35.232 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:35.302 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 10:57:50 AM UTC. 2024-03-29T10:59:35.462 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T10:59:35.463 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:59:35.463 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-03-29T10:59:35.463 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:59:35.463 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T10:59:35.463 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:59:35.463 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:35.463 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T10:59:35.463 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:59:35.463 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-03-29T10:59:35.464 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:35.464 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 3.1 M 2024-03-29T10:59:35.464 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 13 M 2024-03-29T10:59:35.464 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T10:59:35.504 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:35.504 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:35.504 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T10:59:35.504 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:59:35.504 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:35.504 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T10:59:35.725 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs2 2024-03-29T10:59:36.300 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 10:57:56 AM UTC. 2024-03-29T10:59:36.401 INFO:teuthology.orchestra.run.smithi071.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:59:36.455 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T10:59:36.456 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-29T10:59:36.456 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T10:59:36.525 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs-devel 2024-03-29T10:59:37.089 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:57:56 AM UTC. 2024-03-29T10:59:37.240 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T10:59:37.240 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:59:37.241 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-29T10:59:37.241 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:59:37.241 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T10:59:37.241 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:59:37.241 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:37.241 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T10:59:37.241 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:59:37.241 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-29T10:59:37.241 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:37.241 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 31 k 2024-03-29T10:59:37.241 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 140 k 2024-03-29T10:59:37.241 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T10:59:37.400 INFO:teuthology.orchestra.run.smithi071.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 196 kB/s | 31 kB 00:00 2024-03-29T10:59:37.400 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:59:37.401 INFO:teuthology.orchestra.run.smithi071.stdout:Total 195 kB/s | 31 kB 00:00 2024-03-29T10:59:37.401 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T10:59:37.404 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T10:59:37.404 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T10:59:37.430 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T10:59:37.430 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T10:59:37.554 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T10:59:37.657 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:37.662 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 1.4 MB/s | 3.1 MB 00:02 2024-03-29T10:59:37.663 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:59:37.663 INFO:teuthology.orchestra.run.smithi137.stdout:Total 1.4 MB/s | 3.1 MB 00:02 2024-03-29T10:59:37.664 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T10:59:37.674 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T10:59:37.674 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T10:59:37.751 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T10:59:37.751 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T10:59:38.041 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:38.185 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T10:59:38.271 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:38.296 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:38.297 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:59:38.297 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:59:38.297 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:59:38.297 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:59:38.297 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:38.311 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:38.311 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:38.311 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T10:59:38.311 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:59:38.311 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:38.311 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T10:59:38.474 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados2 2024-03-29T10:59:39.034 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 10:57:56 AM UTC. 2024-03-29T10:59:39.130 INFO:teuthology.orchestra.run.smithi071.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:59:39.181 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T10:59:39.182 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-29T10:59:39.182 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T10:59:39.227 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librbd1 2024-03-29T10:59:39.447 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:39.447 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:39.447 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T10:59:39.447 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:59:39.447 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:39.447 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T10:59:39.726 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-nbd 2024-03-29T10:59:39.797 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 10:57:56 AM UTC. 2024-03-29T10:59:39.893 INFO:teuthology.orchestra.run.smithi071.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:59:39.945 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T10:59:39.945 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-29T10:59:39.945 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T10:59:39.991 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rados 2024-03-29T10:59:40.306 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 10:57:50 AM UTC. 2024-03-29T10:59:40.457 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-29T10:59:40.457 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:59:40.457 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-03-29T10:59:40.458 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:59:40.458 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-29T10:59:40.458 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:59:40.458 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:40.458 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-29T10:59:40.458 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-29T10:59:40.458 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-03-29T10:59:40.458 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:40.458 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 171 k 2024-03-29T10:59:40.459 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 494 k 2024-03-29T10:59:40.459 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-29T10:59:40.555 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 10:57:56 AM UTC. 2024-03-29T10:59:40.653 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:59:40.705 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T10:59:40.706 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-29T10:59:40.707 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T10:59:40.752 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rgw 2024-03-29T10:59:40.866 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 422 kB/s | 171 kB 00:00 2024-03-29T10:59:40.866 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:59:40.866 INFO:teuthology.orchestra.run.smithi137.stdout:Total 420 kB/s | 171 kB 00:00 2024-03-29T10:59:40.866 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-29T10:59:40.874 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-29T10:59:40.874 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-29T10:59:40.929 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-29T10:59:40.929 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-29T10:59:41.078 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-29T10:59:41.188 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:41.318 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 10:57:56 AM UTC. 2024-03-29T10:59:41.416 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:59:41.468 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T10:59:41.469 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-29T10:59:41.469 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T10:59:41.512 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-cephfs 2024-03-29T10:59:41.604 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:41.887 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:41.888 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:41.888 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-29T10:59:41.888 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:59:41.888 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:41.888 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-29T10:59:42.079 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 10:57:56 AM UTC. 2024-03-29T10:59:42.110 DEBUG:teuthology.parallel:result is None 2024-03-29T10:59:42.176 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:59:42.228 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T10:59:42.229 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-29T10:59:42.229 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T10:59:42.279 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rbd 2024-03-29T10:59:42.846 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 10:57:56 AM UTC. 2024-03-29T10:59:42.944 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:59:42.996 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T10:59:42.996 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-29T10:59:42.996 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T10:59:43.043 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-fuse 2024-03-29T10:59:43.602 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 10:57:56 AM UTC. 2024-03-29T10:59:43.749 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T10:59:43.750 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:59:43.750 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-03-29T10:59:43.750 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:59:43.750 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T10:59:43.750 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:59:43.750 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:43.751 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T10:59:43.751 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:59:43.751 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-29T10:59:43.751 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:43.751 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 85 k 2024-03-29T10:59:43.751 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 231 k 2024-03-29T10:59:43.751 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T10:59:43.969 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 392 kB/s | 85 kB 00:00 2024-03-29T10:59:43.970 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:59:43.970 INFO:teuthology.orchestra.run.smithi071.stdout:Total 390 kB/s | 85 kB 00:00 2024-03-29T10:59:43.970 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T10:59:43.977 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T10:59:43.977 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T10:59:44.032 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T10:59:44.032 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T10:59:44.169 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T10:59:44.267 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:44.652 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:44.957 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:44.957 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:44.957 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T10:59:44.957 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:59:44.958 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:44.958 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T10:59:45.188 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-mirror 2024-03-29T10:59:45.776 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 10:57:56 AM UTC. 2024-03-29T10:59:45.928 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T10:59:45.929 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:59:45.929 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-29T10:59:45.929 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:59:45.929 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T10:59:45.930 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:59:45.930 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:45.930 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T10:59:45.930 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:59:45.930 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-29T10:59:45.930 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:45.930 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 3.1 M 2024-03-29T10:59:45.930 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 13 M 2024-03-29T10:59:45.930 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T10:59:46.531 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 5.1 MB/s | 3.1 MB 00:00 2024-03-29T10:59:46.531 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:59:46.531 INFO:teuthology.orchestra.run.smithi071.stdout:Total 5.1 MB/s | 3.1 MB 00:00 2024-03-29T10:59:46.532 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T10:59:46.541 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T10:59:46.541 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T10:59:46.614 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T10:59:46.614 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T10:59:47.044 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T10:59:47.098 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:47.123 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:47.123 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:59:47.123 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:59:47.123 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:59:47.123 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:59:47.123 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:48.293 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:48.293 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:48.293 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T10:59:48.293 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:59:48.293 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:48.293 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T10:59:48.542 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-nbd 2024-03-29T10:59:49.117 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 10:57:56 AM UTC. 2024-03-29T10:59:49.265 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T10:59:49.265 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:59:49.266 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-03-29T10:59:49.266 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:59:49.266 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T10:59:49.266 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:59:49.266 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:49.266 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T10:59:49.266 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T10:59:49.266 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-29T10:59:49.266 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:49.266 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 171 k 2024-03-29T10:59:49.266 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 494 k 2024-03-29T10:59:49.266 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T10:59:49.627 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 479 kB/s | 171 kB 00:00 2024-03-29T10:59:49.627 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:59:49.627 INFO:teuthology.orchestra.run.smithi071.stdout:Total 477 kB/s | 171 kB 00:00 2024-03-29T10:59:49.627 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T10:59:49.633 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T10:59:49.633 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T10:59:49.688 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T10:59:49.689 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T10:59:49.840 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T10:59:49.948 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:50.362 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:50.659 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:59:50.659 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:50.659 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T10:59:50.659 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:59:50.659 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:50.659 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T10:59:50.893 DEBUG:teuthology.parallel:result is None 2024-03-29T10:59:50.894 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:59:51.035 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:59:51.055 INFO:teuthology.orchestra.run.smithi071.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:59:51.056 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:59:51.056 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:59:51.057 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:59:51.194 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:59:51.218 INFO:teuthology.orchestra.run.smithi137.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:59:51.219 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:59:51.219 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:59:51.220 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T10:59:51.220 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T10:59:51.220 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:59:51.247 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T10:59:51.248 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:59:51.290 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T10:59:51.290 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T10:59:51.290 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:59:51.317 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:59:51.383 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T10:59:51.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:59:51.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:59:51.482 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T10:59:51.482 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T10:59:51.482 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:59:51.509 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:59:51.583 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T10:59:51.583 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:59:51.612 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:59:51.680 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T10:59:51.680 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T10:59:51.680 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:59:51.707 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:59:51.773 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T10:59:51.773 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:59:51.800 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:59:51.871 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T10:59:51.958 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T10:59:51.958 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 777 /var/log/ceph 2024-03-29T10:59:51.960 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 777 /var/log/ceph 2024-03-29T10:59:51.987 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T10:59:51.987 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:59:52.027 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:59:52.054 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T10:59:52.054 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:59:52.093 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:59:52.130 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T10:59:52.131 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T10:59:52.131 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622670', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622670', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi071.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOcs4iTZhsRuZeUJNRGglRkljVWYqSRuS0hdE/+KYKineMcL/m/NgmcjEif3aIR8bvEZPniQYgZqCuqPgWkrFpw=', 'smithi137.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLfH/EJ2PkJNYZoxy9WM5+7uIdEXJwM+CpijWnhtSm97Q9XamauYkrsr/bGxV6otwT2Zv1yDdYVkkOOT1KktJe0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T10:59:52.131 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:59:52.160 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:59:52.175 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:59:52.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:59:52.243 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T10:59:52.243 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:59:52.295 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:59:52.295 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_1 2024-03-29T10:59:52.349 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:59:52.349 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:59:52.349 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-29T10:59:52.349 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:59:52.350 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:59:52.350 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-29 10:59:47.404718160 +0000 2024-03-29T10:59:52.350 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-29 10:56:52.559452292 +0000 2024-03-29T10:59:52.350 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-29 10:56:52.559452292 +0000 2024-03-29T10:59:52.350 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-29 10:56:52.559452292 +0000 2024-03-29T10:59:52.350 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:59:52.415 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-29T10:59:52.415 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-29T10:59:52.415 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000200653 s, 2.6 MB/s 2024-03-29T10:59:52.416 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:59:52.472 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_2 2024-03-29T10:59:52.526 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:59:52.526 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:59:52.526 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-29T10:59:52.526 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:59:52.527 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:59:52.527 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-29 10:59:47.404718160 +0000 2024-03-29T10:59:52.527 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-29 10:56:52.908446499 +0000 2024-03-29T10:59:52.527 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-29 10:56:52.908446499 +0000 2024-03-29T10:59:52.527 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-29 10:56:52.908446499 +0000 2024-03-29T10:59:52.527 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:59:52.591 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-29T10:59:52.591 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-29T10:59:52.591 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000175805 s, 2.9 MB/s 2024-03-29T10:59:52.592 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:59:52.648 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_3 2024-03-29T10:59:52.702 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:59:52.702 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:59:52.702 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T10:59:52.702 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:59:52.702 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:59:52.703 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-29 10:59:47.405718144 +0000 2024-03-29T10:59:52.703 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-29 10:56:53.255440738 +0000 2024-03-29T10:59:52.703 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-29 10:56:53.255440738 +0000 2024-03-29T10:59:52.703 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-29 10:56:53.255440738 +0000 2024-03-29T10:59:52.703 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:59:52.768 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-29T10:59:52.768 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-29T10:59:52.768 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000211201 s, 2.4 MB/s 2024-03-29T10:59:52.769 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:59:52.824 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_4 2024-03-29T10:59:52.879 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:59:52.879 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:59:52.879 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-29T10:59:52.879 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:59:52.879 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:59:52.879 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-29 10:59:47.405718144 +0000 2024-03-29T10:59:52.879 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-29 10:56:53.590435176 +0000 2024-03-29T10:59:52.879 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-29 10:56:53.590435176 +0000 2024-03-29T10:59:52.880 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-29 10:56:53.590435176 +0000 2024-03-29T10:59:52.880 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:59:52.944 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-29T10:59:52.944 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-29T10:59:52.944 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000183816 s, 2.8 MB/s 2024-03-29T10:59:52.946 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:59:53.001 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:59:53.001 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T10:59:53.001 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:59:53.017 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:59:53.017 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_1 2024-03-29T10:59:53.074 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:59:53.075 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:59:53.075 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-29T10:59:53.075 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:59:53.075 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:59:53.075 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-03-29 10:59:38.586449253 +0000 2024-03-29T10:59:53.075 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-03-29 10:57:06.615562454 +0000 2024-03-29T10:59:53.075 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-03-29 10:57:06.615562454 +0000 2024-03-29T10:59:53.075 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-03-29 10:57:06.615562454 +0000 2024-03-29T10:59:53.075 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:59:53.143 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-03-29T10:59:53.144 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-03-29T10:59:53.144 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000261927 s, 2.0 MB/s 2024-03-29T10:59:53.145 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:59:53.201 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_2 2024-03-29T10:59:53.257 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:59:53.257 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:59:53.257 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-29T10:59:53.257 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:59:53.257 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:59:53.257 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-03-29 10:59:38.586449253 +0000 2024-03-29T10:59:53.257 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-03-29 10:57:06.984562335 +0000 2024-03-29T10:59:53.257 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-03-29 10:57:06.984562335 +0000 2024-03-29T10:59:53.257 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-03-29 10:57:06.984562335 +0000 2024-03-29T10:59:53.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:59:53.323 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-03-29T10:59:53.323 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-03-29T10:59:53.323 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000222598 s, 2.3 MB/s 2024-03-29T10:59:53.324 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:59:53.381 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_3 2024-03-29T10:59:53.436 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:59:53.436 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:59:53.436 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-29T10:59:53.436 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:59:53.436 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:59:53.436 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-03-29 10:59:38.587449252 +0000 2024-03-29T10:59:53.436 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-03-29 10:57:07.334562223 +0000 2024-03-29T10:59:53.437 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-03-29 10:57:07.334562223 +0000 2024-03-29T10:59:53.437 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-03-29 10:57:07.334562223 +0000 2024-03-29T10:59:53.437 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:59:53.501 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-03-29T10:59:53.501 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-03-29T10:59:53.501 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000209378 s, 2.4 MB/s 2024-03-29T10:59:53.506 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:59:53.561 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_4 2024-03-29T10:59:53.615 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:59:53.615 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:59:53.615 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T10:59:53.615 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:59:53.615 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:59:53.616 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-03-29 10:59:38.587449252 +0000 2024-03-29T10:59:53.616 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-03-29 10:57:07.670562115 +0000 2024-03-29T10:59:53.616 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-03-29 10:57:07.670562115 +0000 2024-03-29T10:59:53.616 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-03-29 10:57:07.670562115 +0000 2024-03-29T10:59:53.616 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:59:53.681 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-03-29T10:59:53.681 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-03-29T10:59:53.681 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000188818 s, 2.7 MB/s 2024-03-29T10:59:53.683 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:59:53.739 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:59:53.739 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi137.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:59:53.739 INFO:tasks.ceph:Generating config... 2024-03-29T10:59:53.740 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T10:59:53.740 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T10:59:53.740 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T10:59:53.740 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T10:59:53.740 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T10:59:53.740 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T10:59:53.740 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T10:59:53.740 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T10:59:53.740 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T10:59:53.741 INFO:tasks.ceph:[global] ms type = async 2024-03-29T10:59:53.741 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T10:59:53.741 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T10:59:53.741 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T10:59:53.741 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T10:59:53.741 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T10:59:53.741 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T10:59:53.741 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T10:59:53.741 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T10:59:53.741 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T10:59:53.741 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T10:59:53.741 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T10:59:53.741 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T10:59:53.741 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T10:59:53.741 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-29T10:59:53.741 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T10:59:53.742 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T10:59:53.742 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T10:59:53.742 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T10:59:53.742 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T10:59:53.742 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T10:59:53.742 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T10:59:53.742 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T10:59:53.742 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T10:59:53.742 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T10:59:53.742 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T10:59:53.742 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T10:59:53.742 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T10:59:53.742 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T10:59:53.742 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T10:59:53.743 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T10:59:53.743 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T10:59:53.743 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T10:59:53.743 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T10:59:53.743 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T10:59:53.743 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T10:59:53.743 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T10:59:53.743 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T10:59:53.743 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T10:59:53.743 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T10:59:53.743 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T10:59:53.791 INFO:teuthology.orchestra.run.smithi071.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T10:59:53.793 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T10:59:53.837 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:59:53.863 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.71:6789]'), ('mon.c', '[v1:172.21.15.71:6790]'), ('mon.b', '[v1:172.21.15.137:6789]')] 2024-03-29T10:59:53.863 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.71:6789],[v1:172.21.15.71:6790],[v1:172.21.15.137:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T10:59:53.864 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T10:59:53.864 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:59:53.918 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.71:6789]' --addv c '[v1:172.21.15.71:6790]' --addv b '[v1:172.21.15.137:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:59:54.004 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:59:54.004 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: generated fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T10:59:54.004 INFO:teuthology.orchestra.run.smithi071.stdout:setting min_mon_release = quincy 2024-03-29T10:59:54.004 INFO:teuthology.orchestra.run.smithi071.stdout:epoch 0 2024-03-29T10:59:54.004 INFO:teuthology.orchestra.run.smithi071.stdout:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T10:59:54.004 INFO:teuthology.orchestra.run.smithi071.stdout:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T10:59:54.004 INFO:teuthology.orchestra.run.smithi071.stdout:created 2024-03-29T10:59:54.024963+0000 2024-03-29T10:59:54.004 INFO:teuthology.orchestra.run.smithi071.stdout:min_mon_release 17 (quincy) 2024-03-29T10:59:54.004 INFO:teuthology.orchestra.run.smithi071.stdout:election_strategy: 1 2024-03-29T10:59:54.004 INFO:teuthology.orchestra.run.smithi071.stdout:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T10:59:54.005 INFO:teuthology.orchestra.run.smithi071.stdout:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T10:59:54.005 INFO:teuthology.orchestra.run.smithi071.stdout:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T10:59:54.005 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T10:59:54.006 DEBUG:teuthology.orchestra.run.smithi071:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:59:54.020 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 8c8dcbb1-3a20-411a-b928-492e558bbb2b... 2024-03-29T10:59:54.021 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:59:54.063 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:59:54.110 INFO:teuthology.orchestra.run.smithi071.stdout:[global] 2024-03-29T10:59:54.110 INFO:teuthology.orchestra.run.smithi071.stdout: chdir = "" 2024-03-29T10:59:54.110 INFO:teuthology.orchestra.run.smithi071.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:59:54.110 INFO:teuthology.orchestra.run.smithi071.stdout: auth supported = cephx 2024-03-29T10:59:54.110 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.110 INFO:teuthology.orchestra.run.smithi071.stdout: filestore xattr use omap = true 2024-03-29T10:59:54.111 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.111 INFO:teuthology.orchestra.run.smithi071.stdout: mon clock drift allowed = 1.000 2024-03-29T10:59:54.111 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.111 INFO:teuthology.orchestra.run.smithi071.stdout: osd crush chooseleaf type = 0 2024-03-29T10:59:54.111 INFO:teuthology.orchestra.run.smithi071.stdout: auth debug = true 2024-03-29T10:59:54.111 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.111 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on old message = true 2024-03-29T10:59:54.111 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on bug = true 2024-03-29T10:59:54.111 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.111 INFO:teuthology.orchestra.run.smithi071.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:59:54.111 INFO:teuthology.orchestra.run.smithi071.stdout: mon pg warn max object skew = 0 2024-03-29T10:59:54.111 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.111 INFO:teuthology.orchestra.run.smithi071.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:59:54.111 INFO:teuthology.orchestra.run.smithi071.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:59:54.111 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.112 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default size = 2 2024-03-29T10:59:54.112 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.112 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow primary affinity = true 2024-03-29T10:59:54.112 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow pg remap = true 2024-03-29T10:59:54.112 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:59:54.112 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:59:54.112 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on no sortbitwise = false 2024-03-29T10:59:54.112 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:59:54.112 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on too few osds = false 2024-03-29T10:59:54.112 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:59:54.112 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:59:54.112 INFO:teuthology.orchestra.run.smithi071.stdout: mon_allow_pool_size_one = true 2024-03-29T10:59:54.112 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.112 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:59:54.113 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.113 INFO:teuthology.orchestra.run.smithi071.stdout: osd default data pool replay window = 5 2024-03-29T10:59:54.113 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.113 INFO:teuthology.orchestra.run.smithi071.stdout: mon allow pool delete = true 2024-03-29T10:59:54.113 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.113 INFO:teuthology.orchestra.run.smithi071.stdout: mon cluster log file level = debug 2024-03-29T10:59:54.113 INFO:teuthology.orchestra.run.smithi071.stdout: debug asserts on shutdown = true 2024-03-29T10:59:54.113 INFO:teuthology.orchestra.run.smithi071.stdout: mon health detail to clog = false 2024-03-29T10:59:54.113 INFO:teuthology.orchestra.run.smithi071.stdout: mon host = "[v1:172.21.15.71:6789],[v1:172.21.15.71:6790],[v1:172.21.15.137:6789]" 2024-03-29T10:59:54.113 INFO:teuthology.orchestra.run.smithi071.stdout: lockdep = True 2024-03-29T10:59:54.113 INFO:teuthology.orchestra.run.smithi071.stdout: mon client directed command retry = 5 2024-03-29T10:59:54.113 INFO:teuthology.orchestra.run.smithi071.stdout: mon election default strategy = 1 2024-03-29T10:59:54.113 INFO:teuthology.orchestra.run.smithi071.stdout: ms bind msgr2 = False 2024-03-29T10:59:54.113 INFO:teuthology.orchestra.run.smithi071.stdout: ms inject socket failures = 5000 2024-03-29T10:59:54.113 INFO:teuthology.orchestra.run.smithi071.stdout: ms type = async 2024-03-29T10:59:54.114 INFO:teuthology.orchestra.run.smithi071.stdout: fsid = 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T10:59:54.114 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.114 INFO:teuthology.orchestra.run.smithi071.stdout:[osd] 2024-03-29T10:59:54.114 INFO:teuthology.orchestra.run.smithi071.stdout: osd journal size = 100 2024-03-29T10:59:54.114 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.114 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub load threshold = 5.0 2024-03-29T10:59:54.114 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub max interval = 600 2024-03-29T10:59:54.114 INFO:teuthology.orchestra.run.smithi071.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:59:54.114 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.114 INFO:teuthology.orchestra.run.smithi071.stdout: osd recover clone overlap = true 2024-03-29T10:59:54.114 INFO:teuthology.orchestra.run.smithi071.stdout: osd recovery max chunk = 1048576 2024-03-29T10:59:54.114 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.114 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug shutdown = true 2024-03-29T10:59:54.115 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug op order = true 2024-03-29T10:59:54.115 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify stray on activate = true 2024-03-29T10:59:54.115 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug trim objects = true 2024-03-29T10:59:54.115 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.115 INFO:teuthology.orchestra.run.smithi071.stdout: osd open classes on start = true 2024-03-29T10:59:54.115 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug pg log writeout = true 2024-03-29T10:59:54.115 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.115 INFO:teuthology.orchestra.run.smithi071.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:59:54.118 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.118 INFO:teuthology.orchestra.run.smithi071.stdout: osd map max advance = 10 2024-03-29T10:59:54.118 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.118 INFO:teuthology.orchestra.run.smithi071.stdout: journal zero on create = true 2024-03-29T10:59:54.118 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.118 INFO:teuthology.orchestra.run.smithi071.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:59:54.118 INFO:teuthology.orchestra.run.smithi071.stdout: filestore apply finisher threads = 3 2024-03-29T10:59:54.118 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.118 INFO:teuthology.orchestra.run.smithi071.stdout: bdev debug aio = true 2024-03-29T10:59:54.118 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug misdirected ops = true 2024-03-29T10:59:54.119 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore block size = 96636764160 2024-03-29T10:59:54.119 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore compression algorithm = zstd 2024-03-29T10:59:54.119 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore compression mode = aggressive 2024-03-29T10:59:54.119 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore fsck on mount = True 2024-03-29T10:59:54.119 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore zero block detection = True 2024-03-29T10:59:54.119 INFO:teuthology.orchestra.run.smithi071.stdout: debug bluefs = 20 2024-03-29T10:59:54.119 INFO:teuthology.orchestra.run.smithi071.stdout: debug bluestore = 20 2024-03-29T10:59:54.119 INFO:teuthology.orchestra.run.smithi071.stdout: debug monc = 20 2024-03-29T10:59:54.119 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-03-29T10:59:54.119 INFO:teuthology.orchestra.run.smithi071.stdout: debug osd = 20 2024-03-29T10:59:54.119 INFO:teuthology.orchestra.run.smithi071.stdout: debug rocksdb = 10 2024-03-29T10:59:54.119 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:59:54.119 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd full ratio = 0.9 2024-03-29T10:59:54.119 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:59:54.120 INFO:teuthology.orchestra.run.smithi071.stdout: osd class default list = * 2024-03-29T10:59:54.120 INFO:teuthology.orchestra.run.smithi071.stdout: osd class load list = * 2024-03-29T10:59:54.120 INFO:teuthology.orchestra.run.smithi071.stdout: osd client watch timeout = 120 2024-03-29T10:59:54.120 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify cached snaps = True 2024-03-29T10:59:54.120 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify missing on start = True 2024-03-29T10:59:54.120 INFO:teuthology.orchestra.run.smithi071.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:59:54.120 INFO:teuthology.orchestra.run.smithi071.stdout: osd mclock override recovery settings = True 2024-03-29T10:59:54.120 INFO:teuthology.orchestra.run.smithi071.stdout: osd objectstore = bluestore 2024-03-29T10:59:54.120 INFO:teuthology.orchestra.run.smithi071.stdout: osd op queue = debug_random 2024-03-29T10:59:54.120 INFO:teuthology.orchestra.run.smithi071.stdout: osd op queue cut off = debug_random 2024-03-29T10:59:54.120 INFO:teuthology.orchestra.run.smithi071.stdout: osd shutdown pgref assert = True 2024-03-29T10:59:54.120 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.121 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr] 2024-03-29T10:59:54.121 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-03-29T10:59:54.121 INFO:teuthology.orchestra.run.smithi071.stdout: debug mgr = 20 2024-03-29T10:59:54.121 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2024-03-29T10:59:54.121 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2024-03-29T10:59:54.121 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:59:54.121 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:59:54.121 INFO:teuthology.orchestra.run.smithi071.stdout: mgr/telemetry/nag = false 2024-03-29T10:59:54.121 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.121 INFO:teuthology.orchestra.run.smithi071.stdout:[mon] 2024-03-29T10:59:54.121 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-03-29T10:59:54.121 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2024-03-29T10:59:54.121 INFO:teuthology.orchestra.run.smithi071.stdout: debug paxos = 20 2024-03-29T10:59:54.121 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2024-03-29T10:59:54.122 INFO:teuthology.orchestra.run.smithi071.stdout: mon data avail warn = 5 2024-03-29T10:59:54.122 INFO:teuthology.orchestra.run.smithi071.stdout: mon mgr mkfs grace = 240 2024-03-29T10:59:54.122 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:59:54.122 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd reporter subtree level = osd 2024-03-29T10:59:54.122 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd prime pg temp = true 2024-03-29T10:59:54.122 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:59:54.122 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.122 INFO:teuthology.orchestra.run.smithi071.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:59:54.122 INFO:teuthology.orchestra.run.smithi071.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:59:54.122 INFO:teuthology.orchestra.run.smithi071.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:59:54.122 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.122 INFO:teuthology.orchestra.run.smithi071.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:59:54.122 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:59:54.123 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:59:54.123 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.123 INFO:teuthology.orchestra.run.smithi071.stdout: # 1m isn't quite enough 2024-03-29T10:59:54.123 INFO:teuthology.orchestra.run.smithi071.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:59:54.123 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.123 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:59:54.123 INFO:teuthology.orchestra.run.smithi071.stdout: mon min osdmap epochs = 50 2024-03-29T10:59:54.123 INFO:teuthology.orchestra.run.smithi071.stdout: mon osdmap full prune interval = 2 2024-03-29T10:59:54.123 INFO:teuthology.orchestra.run.smithi071.stdout: mon osdmap full prune min = 15 2024-03-29T10:59:54.123 INFO:teuthology.orchestra.run.smithi071.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:59:54.123 INFO:teuthology.orchestra.run.smithi071.stdout: mon scrub interval = 300 2024-03-29T10:59:54.123 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on pool no app = False 2024-03-29T10:59:54.123 INFO:teuthology.orchestra.run.smithi071.stdout: paxos service trim min = 10 2024-03-29T10:59:54.123 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T10:59:54.124 INFO:teuthology.orchestra.run.smithi071.stdout:[client] 2024-03-29T10:59:54.124 INFO:teuthology.orchestra.run.smithi071.stdout: rgw cache enabled = true 2024-03-29T10:59:54.124 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable ops log = true 2024-03-29T10:59:54.124 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable usage log = true 2024-03-29T10:59:54.124 INFO:teuthology.orchestra.run.smithi071.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:59:54.124 INFO:teuthology.orchestra.run.smithi071.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:59:54.124 INFO:teuthology.orchestra.run.smithi071.stdout: debug monc = 20 2024-03-29T10:59:54.124 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-03-29T10:59:54.124 INFO:teuthology.orchestra.run.smithi071.stdout: debug objecter = 20 2024-03-29T10:59:54.124 INFO:teuthology.orchestra.run.smithi071.stdout: debug rados = 20 2024-03-29T10:59:54.124 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.a] 2024-03-29T10:59:54.124 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.c] 2024-03-29T10:59:54.124 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.b] 2024-03-29T10:59:54.125 INFO:teuthology.orchestra.run.smithi137.stdout:[global] 2024-03-29T10:59:54.125 INFO:teuthology.orchestra.run.smithi137.stdout: chdir = "" 2024-03-29T10:59:54.125 INFO:teuthology.orchestra.run.smithi137.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:59:54.125 INFO:teuthology.orchestra.run.smithi137.stdout: auth supported = cephx 2024-03-29T10:59:54.125 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.125 INFO:teuthology.orchestra.run.smithi137.stdout: filestore xattr use omap = true 2024-03-29T10:59:54.125 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.125 INFO:teuthology.orchestra.run.smithi137.stdout: mon clock drift allowed = 1.000 2024-03-29T10:59:54.125 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.125 INFO:teuthology.orchestra.run.smithi137.stdout: osd crush chooseleaf type = 0 2024-03-29T10:59:54.126 INFO:teuthology.orchestra.run.smithi137.stdout: auth debug = true 2024-03-29T10:59:54.126 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.126 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on old message = true 2024-03-29T10:59:54.126 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on bug = true 2024-03-29T10:59:54.126 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.126 INFO:teuthology.orchestra.run.smithi137.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:59:54.126 INFO:teuthology.orchestra.run.smithi137.stdout: mon pg warn max object skew = 0 2024-03-29T10:59:54.126 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.126 INFO:teuthology.orchestra.run.smithi137.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:59:54.126 INFO:teuthology.orchestra.run.smithi137.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:59:54.126 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.126 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default size = 2 2024-03-29T10:59:54.126 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.126 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow primary affinity = true 2024-03-29T10:59:54.126 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow pg remap = true 2024-03-29T10:59:54.126 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:59:54.127 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:59:54.127 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on no sortbitwise = false 2024-03-29T10:59:54.127 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:59:54.127 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on too few osds = false 2024-03-29T10:59:54.127 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:59:54.127 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:59:54.127 INFO:teuthology.orchestra.run.smithi137.stdout: mon_allow_pool_size_one = true 2024-03-29T10:59:54.127 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.127 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:59:54.127 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.127 INFO:teuthology.orchestra.run.smithi137.stdout: osd default data pool replay window = 5 2024-03-29T10:59:54.127 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.127 INFO:teuthology.orchestra.run.smithi137.stdout: mon allow pool delete = true 2024-03-29T10:59:54.127 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.127 INFO:teuthology.orchestra.run.smithi137.stdout: mon cluster log file level = debug 2024-03-29T10:59:54.127 INFO:teuthology.orchestra.run.smithi137.stdout: debug asserts on shutdown = true 2024-03-29T10:59:54.127 INFO:teuthology.orchestra.run.smithi137.stdout: mon health detail to clog = false 2024-03-29T10:59:54.128 INFO:teuthology.orchestra.run.smithi137.stdout: mon host = "[v1:172.21.15.71:6789],[v1:172.21.15.71:6790],[v1:172.21.15.137:6789]" 2024-03-29T10:59:54.128 INFO:teuthology.orchestra.run.smithi137.stdout: lockdep = True 2024-03-29T10:59:54.128 INFO:teuthology.orchestra.run.smithi137.stdout: mon client directed command retry = 5 2024-03-29T10:59:54.128 INFO:teuthology.orchestra.run.smithi137.stdout: mon election default strategy = 1 2024-03-29T10:59:54.128 INFO:teuthology.orchestra.run.smithi137.stdout: ms bind msgr2 = False 2024-03-29T10:59:54.128 INFO:teuthology.orchestra.run.smithi137.stdout: ms inject socket failures = 5000 2024-03-29T10:59:54.128 INFO:teuthology.orchestra.run.smithi137.stdout: ms type = async 2024-03-29T10:59:54.128 INFO:teuthology.orchestra.run.smithi137.stdout: fsid = 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T10:59:54.128 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.128 INFO:teuthology.orchestra.run.smithi137.stdout:[osd] 2024-03-29T10:59:54.128 INFO:teuthology.orchestra.run.smithi137.stdout: osd journal size = 100 2024-03-29T10:59:54.128 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.128 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub load threshold = 5.0 2024-03-29T10:59:54.128 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub max interval = 600 2024-03-29T10:59:54.129 INFO:teuthology.orchestra.run.smithi137.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:59:54.129 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.129 INFO:teuthology.orchestra.run.smithi137.stdout: osd recover clone overlap = true 2024-03-29T10:59:54.129 INFO:teuthology.orchestra.run.smithi137.stdout: osd recovery max chunk = 1048576 2024-03-29T10:59:54.129 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.129 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug shutdown = true 2024-03-29T10:59:54.129 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug op order = true 2024-03-29T10:59:54.129 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug verify stray on activate = true 2024-03-29T10:59:54.129 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug trim objects = true 2024-03-29T10:59:54.129 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.129 INFO:teuthology.orchestra.run.smithi137.stdout: osd open classes on start = true 2024-03-29T10:59:54.129 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug pg log writeout = true 2024-03-29T10:59:54.129 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.129 INFO:teuthology.orchestra.run.smithi137.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:59:54.129 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: osd map max advance = 10 2024-03-29T10:59:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: journal zero on create = true 2024-03-29T10:59:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:59:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: filestore apply finisher threads = 3 2024-03-29T10:59:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: bdev debug aio = true 2024-03-29T10:59:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug misdirected ops = true 2024-03-29T10:59:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: bluestore block size = 96636764160 2024-03-29T10:59:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: bluestore compression algorithm = zstd 2024-03-29T10:59:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: bluestore compression mode = aggressive 2024-03-29T10:59:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: bluestore fsck on mount = True 2024-03-29T10:59:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: bluestore zero block detection = True 2024-03-29T10:59:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: debug bluefs = 20 2024-03-29T10:59:54.130 INFO:teuthology.orchestra.run.smithi137.stdout: debug bluestore = 20 2024-03-29T10:59:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: debug monc = 20 2024-03-29T10:59:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-03-29T10:59:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: debug osd = 20 2024-03-29T10:59:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: debug rocksdb = 10 2024-03-29T10:59:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:59:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd full ratio = 0.9 2024-03-29T10:59:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:59:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: osd class default list = * 2024-03-29T10:59:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: osd class load list = * 2024-03-29T10:59:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: osd client watch timeout = 120 2024-03-29T10:59:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug verify cached snaps = True 2024-03-29T10:59:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug verify missing on start = True 2024-03-29T10:59:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:59:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: osd mclock override recovery settings = True 2024-03-29T10:59:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: osd objectstore = bluestore 2024-03-29T10:59:54.131 INFO:teuthology.orchestra.run.smithi137.stdout: osd op queue = debug_random 2024-03-29T10:59:54.132 INFO:teuthology.orchestra.run.smithi137.stdout: osd op queue cut off = debug_random 2024-03-29T10:59:54.132 INFO:teuthology.orchestra.run.smithi137.stdout: osd shutdown pgref assert = True 2024-03-29T10:59:54.132 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.132 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr] 2024-03-29T10:59:54.132 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-03-29T10:59:54.132 INFO:teuthology.orchestra.run.smithi137.stdout: debug mgr = 20 2024-03-29T10:59:54.132 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2024-03-29T10:59:54.132 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2024-03-29T10:59:54.132 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:59:54.132 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:59:54.132 INFO:teuthology.orchestra.run.smithi137.stdout: mgr/telemetry/nag = false 2024-03-29T10:59:54.132 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.132 INFO:teuthology.orchestra.run.smithi137.stdout:[mon] 2024-03-29T10:59:54.132 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-03-29T10:59:54.132 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2024-03-29T10:59:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: debug paxos = 20 2024-03-29T10:59:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2024-03-29T10:59:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: mon data avail warn = 5 2024-03-29T10:59:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: mon mgr mkfs grace = 240 2024-03-29T10:59:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:59:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd reporter subtree level = osd 2024-03-29T10:59:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd prime pg temp = true 2024-03-29T10:59:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:59:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:59:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:59:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:59:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:59:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:59:54.133 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:59:54.134 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.134 INFO:teuthology.orchestra.run.smithi137.stdout: # 1m isn't quite enough 2024-03-29T10:59:54.134 INFO:teuthology.orchestra.run.smithi137.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:59:54.134 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.134 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:59:54.134 INFO:teuthology.orchestra.run.smithi137.stdout: mon min osdmap epochs = 50 2024-03-29T10:59:54.134 INFO:teuthology.orchestra.run.smithi137.stdout: mon osdmap full prune interval = 2 2024-03-29T10:59:54.134 INFO:teuthology.orchestra.run.smithi137.stdout: mon osdmap full prune min = 15 2024-03-29T10:59:54.134 INFO:teuthology.orchestra.run.smithi137.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:59:54.134 INFO:teuthology.orchestra.run.smithi137.stdout: mon scrub interval = 300 2024-03-29T10:59:54.134 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on pool no app = False 2024-03-29T10:59:54.134 INFO:teuthology.orchestra.run.smithi137.stdout: paxos service trim min = 10 2024-03-29T10:59:54.134 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-29T10:59:54.134 INFO:teuthology.orchestra.run.smithi137.stdout:[client] 2024-03-29T10:59:54.134 INFO:teuthology.orchestra.run.smithi137.stdout: rgw cache enabled = true 2024-03-29T10:59:54.134 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable ops log = true 2024-03-29T10:59:54.135 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable usage log = true 2024-03-29T10:59:54.135 INFO:teuthology.orchestra.run.smithi137.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:59:54.135 INFO:teuthology.orchestra.run.smithi137.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:59:54.135 INFO:teuthology.orchestra.run.smithi137.stdout: debug monc = 20 2024-03-29T10:59:54.135 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-03-29T10:59:54.135 INFO:teuthology.orchestra.run.smithi137.stdout: debug objecter = 20 2024-03-29T10:59:54.135 INFO:teuthology.orchestra.run.smithi137.stdout: debug rados = 20 2024-03-29T10:59:54.135 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.a] 2024-03-29T10:59:54.135 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.c] 2024-03-29T10:59:54.135 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.b] 2024-03-29T10:59:54.136 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T10:59:54.137 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T10:59:54.191 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T10:59:54.192 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T10:59:54.192 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T10:59:54.206 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T10:59:54.206 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T10:59:54.261 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi071.front.sepia.ceph.com 2024-03-29T10:59:54.261 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T10:59:54.262 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:59:54.262 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:59:54.340 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T10:59:54.341 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:59:54.395 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi137.front.sepia.ceph.com 2024-03-29T10:59:54.395 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T10:59:54.395 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:59:54.395 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:59:54.436 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T10:59:54.436 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:59:54.491 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T10:59:54.492 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T10:59:54.492 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:59:54.546 INFO:teuthology.orchestra.run.smithi071.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:59:54.548 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:59:54.619 INFO:teuthology.orchestra.run.smithi137.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:59:54.621 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T10:59:54.621 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T10:59:54.622 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T10:59:54.663 INFO:teuthology.orchestra.run.smithi071.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T10:59:54.678 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T10:59:54.720 INFO:teuthology.orchestra.run.smithi137.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T10:59:54.736 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T10:59:54.736 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi137.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:59:54.736 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T10:59:54.762 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:59:54.762 INFO:tasks.ceph:role: osd.0 2024-03-29T10:59:54.762 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com 2024-03-29T10:59:54.762 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:59:54.836 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:59:54.837 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:59:54.837 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:59:54.837 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:59:54.837 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:59:54.837 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:59:54.837 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:59:54.837 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:59:54.837 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:59:54.837 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:59:55.350 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-03-29T10:59:55.352 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-03-29T10:59:55.352 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T10:59:55.489 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T10:59:55.523 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T10:59:55.590 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:59:55.590 INFO:tasks.ceph:role: osd.1 2024-03-29T10:59:55.590 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com 2024-03-29T10:59:55.591 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:59:55.664 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:59:55.664 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:59:55.664 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:59:55.664 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:59:55.664 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:59:55.664 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:59:55.664 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:59:55.664 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:59:55.664 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:59:55.664 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:59:56.110 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-03-29T10:59:56.112 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-03-29T10:59:56.113 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T10:59:56.209 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T10:59:56.240 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T10:59:56.306 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:59:56.306 INFO:tasks.ceph:role: osd.2 2024-03-29T10:59:56.306 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi071.front.sepia.ceph.com 2024-03-29T10:59:56.306 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:59:56.380 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:59:56.380 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:59:56.380 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:59:56.380 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:59:56.380 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:59:56.380 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:59:56.380 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:59:56.381 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:59:56.381 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:59:56.381 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:59:56.862 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-03-29T10:59:56.863 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-03-29T10:59:56.864 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T10:59:57.012 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T10:59:57.044 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T10:59:57.111 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:59:57.111 INFO:tasks.ceph:role: osd.3 2024-03-29T10:59:57.112 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi071.front.sepia.ceph.com 2024-03-29T10:59:57.112 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:59:57.185 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:59:57.185 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:59:57.185 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:59:57.185 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:59:57.186 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:59:57.186 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:59:57.186 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:59:57.186 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:59:57.186 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:59:57.186 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:59:57.717 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-03-29T10:59:57.719 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-03-29T10:59:57.719 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T10:59:57.852 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T10:59:57.885 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:59:57.980 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T10:59:58.000+0000 7f565500b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T10:59:57.980 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T10:59:58.000+0000 7f565500b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T10:59:57.980 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T10:59:58.000+0000 7f565500b640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:59:57.980 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T10:59:58.000+0000 7f565500b640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:59:57.981 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T10:59:58.000+0000 7f565500b640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:59:57.981 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T10:59:58.001+0000 7f565500b640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:00:00.829 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:00:00.860 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:00:00.951 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:00.970+0000 7fcf3216f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:00:00.951 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:00.971+0000 7fcf3216f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:00:00.951 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:00.971+0000 7fcf3216f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:00:00.951 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:00.971+0000 7fcf3216f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:00:00.951 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:00.971+0000 7fcf3216f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:00:00.951 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:00.971+0000 7fcf3216f640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:00:03.791 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:00:03.822 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:00:03.913 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:03.932+0000 7f6270c15640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:00:03.913 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:03.932+0000 7f6270c15640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:00:03.913 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:03.932+0000 7f6270c15640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:00:03.913 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:03.932+0000 7f6270c15640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:00:03.913 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:03.932+0000 7f6270c15640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:00:03.914 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:03.933+0000 7f6270c15640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:00:06.768 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:00:06.801 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:00:06.895 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:06.915+0000 7f02ae22a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:00:06.895 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:06.915+0000 7f02ae22a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:00:06.895 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:06.915+0000 7f02ae22a640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:00:06.896 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:06.915+0000 7f02ae22a640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:00:06.896 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:06.915+0000 7f02ae22a640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:00:06.896 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:06.916+0000 7f02ae22a640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:00:09.748 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:00:09.778 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:00:09.813 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:00:09.813 INFO:tasks.ceph:role: osd.4 2024-03-29T11:00:09.813 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi137.front.sepia.ceph.com 2024-03-29T11:00:09.813 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:00:09.890 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:00:09.890 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:00:09.890 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:00:09.890 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:00:09.890 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:00:09.890 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:00:09.890 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:00:09.890 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:00:09.890 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:00:09.890 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:00:10.403 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-03-29T11:00:10.405 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-03-29T11:00:10.405 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:00:10.547 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:00:10.584 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:00:10.653 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:00:10.654 INFO:tasks.ceph:role: osd.5 2024-03-29T11:00:10.654 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi137.front.sepia.ceph.com 2024-03-29T11:00:10.654 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:00:10.729 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:00:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:00:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:00:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:00:10.729 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:00:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:00:10.729 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:00:10.729 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:00:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:00:10.729 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:00:11.174 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-03-29T11:00:11.177 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-03-29T11:00:11.177 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:00:11.268 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:00:11.301 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:00:11.370 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:00:11.370 INFO:tasks.ceph:role: osd.6 2024-03-29T11:00:11.370 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi137.front.sepia.ceph.com 2024-03-29T11:00:11.370 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:00:11.445 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:00:11.446 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:00:11.446 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:00:11.446 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:00:11.446 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:00:11.446 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:00:11.446 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:00:11.446 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:00:11.446 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:00:11.446 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:00:11.955 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-03-29T11:00:11.958 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-03-29T11:00:11.958 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:00:12.107 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:00:12.143 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:00:12.212 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:00:12.212 INFO:tasks.ceph:role: osd.7 2024-03-29T11:00:12.213 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi137.front.sepia.ceph.com 2024-03-29T11:00:12.213 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:00:12.289 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:00:12.289 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:00:12.289 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:00:12.289 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:00:12.289 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:00:12.289 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:00:12.289 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:00:12.289 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:00:12.289 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:00:12.290 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:00:12.792 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-03-29T11:00:12.794 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-03-29T11:00:12.794 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:00:12.919 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:00:12.953 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:00:13.053 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:13.072+0000 7f74f641b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:00:13.053 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:13.073+0000 7f74f641b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:00:13.053 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:13.073+0000 7f74f641b640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:00:13.053 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:13.073+0000 7f74f641b640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:00:13.053 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:13.073+0000 7f74f641b640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:00:13.053 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:13.073+0000 7f74f641b640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:00:15.907 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:00:15.939 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:00:16.031 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:16.050+0000 7f05eb268640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:00:16.031 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:16.050+0000 7f05eb268640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:00:16.031 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:16.050+0000 7f05eb268640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:00:16.031 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:16.050+0000 7f05eb268640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:00:16.031 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:16.050+0000 7f05eb268640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:00:16.031 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:16.051+0000 7f05eb268640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:00:18.873 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:00:18.904 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:00:18.996 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:19.016+0000 7f1d1ac13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:00:18.996 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:19.016+0000 7f1d1ac13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:00:18.996 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:19.016+0000 7f1d1ac13640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:00:18.997 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:19.016+0000 7f1d1ac13640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:00:18.997 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:19.016+0000 7f1d1ac13640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:00:18.997 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:19.017+0000 7f1d1ac13640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:00:21.844 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:00:21.875 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:00:21.966 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:21.986+0000 7f26fb59a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:00:21.967 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:21.986+0000 7f26fb59a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:00:21.967 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:21.986+0000 7f26fb59a640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:00:21.967 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:21.986+0000 7f26fb59a640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:00:21.967 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:21.986+0000 7f26fb59a640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:00:21.967 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:21.987+0000 7f26fb59a640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:00:24.814 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:00:24.843 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:00:24.843 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:00:24.844 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:00:24.876 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:00:24.876 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:00:24.946 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:00:24.946 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:00:25.014 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:00:25.014 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:00:25.082 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:00:25.082 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:00:25.150 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T11:00:25.151 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:00:25.181 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T11:00:25.181 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:00:25.250 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T11:00:25.250 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:00:25.318 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T11:00:25.318 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:00:25.385 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T11:00:25.386 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:00:25.453 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:00:25.453 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:00:25.468 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T11:00:25.468 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:00:25.509 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:00:25.509 DEBUG:teuthology.orchestra.run.smithi071:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:00:25.511 DEBUG:teuthology.orchestra.run.smithi137:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:00:25.552 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr.y] 2024-03-29T11:00:25.552 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAqnwZm7E/SIRAAYWgy668FR7Y8sDqwIenfxQ== 2024-03-29T11:00:25.552 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.0] 2024-03-29T11:00:25.552 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAunwZm9bQTABAAl4MRjUc3Jz7yxX6zlsVqvQ== 2024-03-29T11:00:25.552 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.1] 2024-03-29T11:00:25.552 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAwnwZmnIDpORAAVDeJ3azQSEuYGtZ8SGjECA== 2024-03-29T11:00:25.552 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.2] 2024-03-29T11:00:25.552 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAznwZmPxCgNxAAMM6RqbIQuPQUF244PXPKPg== 2024-03-29T11:00:25.552 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.3] 2024-03-29T11:00:25.553 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQA2nwZmrE2ZNhAAwKGFpYpBgyOzPAo6ENN2Ng== 2024-03-29T11:00:25.553 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr.x] 2024-03-29T11:00:25.553 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAqnwZmd0coJhAAW/aPf767jTxrrVtjXEQiYg== 2024-03-29T11:00:25.553 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.4] 2024-03-29T11:00:25.553 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQA9nwZm9cFmBBAAH0x5+9852qwixo8+gXNaIQ== 2024-03-29T11:00:25.553 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.5] 2024-03-29T11:00:25.553 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQBAnwZmkvISAxAA14ZLFEIt4CJ9sXpT2MyWlw== 2024-03-29T11:00:25.553 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.6] 2024-03-29T11:00:25.553 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQBDnwZmlhsKARAAsUcc6XCFA0mw5L/N49pD6g== 2024-03-29T11:00:25.553 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.7] 2024-03-29T11:00:25.553 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQBFnwZm3zjfOhAA7RMmIcu9AeV95nzRmWZOhg== 2024-03-29T11:00:25.553 INFO:teuthology.orchestra.run.smithi071.stdout:[client.0] 2024-03-29T11:00:25.553 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAqnwZmOZzOKBAAA9ySeY4EHojjtAibAzc8/A== 2024-03-29T11:00:25.554 INFO:teuthology.orchestra.run.smithi071.stdout:[client.1] 2024-03-29T11:00:25.554 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAqnwZmUPc0LBAAtabjE1K0kLIeJYSOQUs7lg== 2024-03-29T11:00:25.574 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr.y] 2024-03-29T11:00:25.574 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQAqnwZm7E/SIRAAYWgy668FR7Y8sDqwIenfxQ== 2024-03-29T11:00:25.574 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.0] 2024-03-29T11:00:25.574 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQAunwZm9bQTABAAl4MRjUc3Jz7yxX6zlsVqvQ== 2024-03-29T11:00:25.574 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.1] 2024-03-29T11:00:25.574 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQAwnwZmnIDpORAAVDeJ3azQSEuYGtZ8SGjECA== 2024-03-29T11:00:25.574 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.2] 2024-03-29T11:00:25.575 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQAznwZmPxCgNxAAMM6RqbIQuPQUF244PXPKPg== 2024-03-29T11:00:25.575 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.3] 2024-03-29T11:00:25.575 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQA2nwZmrE2ZNhAAwKGFpYpBgyOzPAo6ENN2Ng== 2024-03-29T11:00:25.575 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr.x] 2024-03-29T11:00:25.575 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQAqnwZmd0coJhAAW/aPf767jTxrrVtjXEQiYg== 2024-03-29T11:00:25.575 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.4] 2024-03-29T11:00:25.575 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQA9nwZm9cFmBBAAH0x5+9852qwixo8+gXNaIQ== 2024-03-29T11:00:25.575 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.5] 2024-03-29T11:00:25.575 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQBAnwZmkvISAxAA14ZLFEIt4CJ9sXpT2MyWlw== 2024-03-29T11:00:25.575 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.6] 2024-03-29T11:00:25.575 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQBDnwZmlhsKARAAsUcc6XCFA0mw5L/N49pD6g== 2024-03-29T11:00:25.575 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.7] 2024-03-29T11:00:25.575 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQBFnwZm3zjfOhAA7RMmIcu9AeV95nzRmWZOhg== 2024-03-29T11:00:25.575 INFO:teuthology.orchestra.run.smithi137.stdout:[client.0] 2024-03-29T11:00:25.577 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQAqnwZmOZzOKBAAA9ySeY4EHojjtAibAzc8/A== 2024-03-29T11:00:25.577 INFO:teuthology.orchestra.run.smithi137.stdout:[client.1] 2024-03-29T11:00:25.577 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQAqnwZmUPc0LBAAtabjE1K0kLIeJYSOQUs7lg== 2024-03-29T11:00:25.578 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:00:25.597 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:00:25.669 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:00:25.671 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:00:25.723 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:00:25.725 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:00:25.776 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:00:25.778 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:00:25.829 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:00:25.832 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:00:25.883 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:00:25.885 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:00:25.936 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:00:25.938 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:00:25.991 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:00:25.993 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:00:26.045 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:00:26.047 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:00:26.099 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:00:26.101 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:00:26.152 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:00:26.154 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:00:26.210 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:00:26.212 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:00:26.268 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:00:26.268 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:00:26.294 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:00:26.391 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:00:26.418 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:00:26.484 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:00:26.580 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:00:26.607 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:00:26.635 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:00:26.733 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:00:26.760 DEBUG:teuthology.orchestra.run.smithi071:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:00:26.762 DEBUG:teuthology.orchestra.run.smithi137:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:00:26.814 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:00:26.815 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:00:26.815 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:00:26.817 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:00:26.817 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:00:26.817 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:00:26.819 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:00:26.819 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:00:26.819 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:00:26.857 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:00:26.857 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:00:26.857 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:00:26.858 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:00:26.860 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:00:26.860 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:00:26.860 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:00:26.862 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:00:26.862 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:00:26.862 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:00:26.963 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:26.983+0000 7f823a657640 10 monclient: get_monmap_and_config 2024-03-29T11:00:26.963 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:26.983+0000 7f823a657640 10 monclient: build_initial_monmap 2024-03-29T11:00:26.963 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:26.983+0000 7f823a657640 10 monclient: monmap: 2024-03-29T11:00:26.963 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-03-29T11:00:26.963 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:26.964 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-29T11:00:26.984449+0000 2024-03-29T11:00:26.964 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-29T11:00:26.984449+0000 2024-03-29T11:00:26.964 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:26.964 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:26.964 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:26.964 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:00:26.964 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:00:26.964 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:00:26.965 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:26.984+0000 7f823a657640 1 Processor -- start 2024-03-29T11:00:26.965 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:26.984+0000 7f823a657640 1 -- start start 2024-03-29T11:00:26.965 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:26.984+0000 7f823a657640 10 monclient: init 2024-03-29T11:00:26.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:26.984+0000 7f823a657640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:26.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:26.984+0000 7f823a657640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:00:26.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:26.984+0000 7f823a657640 10 monclient(hunting): picked mon.noname-a con 0x7f823409ac40 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:26.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:26.985+0000 7f823a657640 10 monclient(hunting): picked mon.noname-b con 0x7f823409c870 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:26.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:26.985+0000 7f823a657640 10 monclient(hunting): picked mon.noname-c con 0x7f82340b4360 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:26.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:26.985+0000 7f823a657640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f823409dec0 con 0x7f823409ac40 2024-03-29T11:00:26.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:26.985+0000 7f823a657640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f823409da30 con 0x7f82340b4360 2024-03-29T11:00:26.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:26.985+0000 7f823a657640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82340b39c0 con 0x7f823409c870 2024-03-29T11:00:26.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:26.985+0000 7f823a657640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:26.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:26.985+0000 7f823a657640 10 monclient(hunting): authenticate will time out at 783.243155s 2024-03-29T11:00:26.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:26.985+0000 7f82337fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f823409c870 0x7f823413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:38212/0 (socket says 172.21.15.71:38212) 2024-03-29T11:00:26.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:26.985+0000 7f82337fe640 1 -- 172.21.15.71:0/2556991170 learned_addr learned my addr 172.21.15.71:0/2556991170 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:00:26.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:26.985+0000 7f8233fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f823409ac40 0x7f82340b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:60306/0 (socket says 172.21.15.71:60306) 2024-03-29T11:00:29.965 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:29.985+0000 7f8231ffb640 10 monclient(hunting): tick 2024-03-29T11:00:29.965 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:29.985+0000 7f8231ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:00:29.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:29.985+0000 7f8231ffb640 1 monclient(hunting): continuing hunt 2024-03-29T11:00:29.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:29.985+0000 7f8231ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:00:29.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:29.985+0000 7f8231ffb640 1 -- 172.21.15.71:0/2556991170 >> v1:172.21.15.71:6790/0 conn(0x7f823409c870 legacy=0x7f823413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:29.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:29.985+0000 7f8231ffb640 1 -- 172.21.15.71:0/2556991170 >> v1:172.21.15.137:6789/0 conn(0x7f82340b4360 legacy=0x7f823414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:29.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:29.985+0000 7f8231ffb640 1 -- 172.21.15.71:0/2556991170 >> v1:172.21.15.71:6789/0 conn(0x7f823409ac40 legacy=0x7f82340b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:29.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:29.985+0000 7f8231ffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:00:29.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:29.985+0000 7f8231ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f82200036e0 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:29.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:29.985+0000 7f8231ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f8220007230 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:29.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:29.985+0000 7f8231ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f822000af00 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:29.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:29.985+0000 7f8231ffb640 1 -- 172.21.15.71:0/2556991170 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82340b5160 con 0x7f8220007230 2024-03-29T11:00:29.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:29.985+0000 7f8231ffb640 1 -- 172.21.15.71:0/2556991170 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f822000fbd0 con 0x7f822000af00 2024-03-29T11:00:29.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:29.985+0000 7f8231ffb640 1 -- 172.21.15.71:0/2556991170 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8220010e30 con 0x7f82200036e0 2024-03-29T11:00:29.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:29.985+0000 7f8231ffb640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:32.135 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.154+0000 7f8232ffd640 1 -- 172.21.15.71:0/2556991170 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227432287 0 0) 0x7f82340b5160 con 0x7f8220007230 2024-03-29T11:00:32.135 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.154+0000 7f8232ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:32.135 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.154+0000 7f8232ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:32.135 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.154+0000 7f8232ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:32.135 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.154+0000 7f8232ffd640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:00:32.135 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.154+0000 7f8232ffd640 1 -- 172.21.15.71:0/2556991170 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f821c003520 con 0x7f8220007230 2024-03-29T11:00:32.137 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.156+0000 7f8232ffd640 1 -- 172.21.15.71:0/2556991170 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3784976239 0 0) 0x7f821c003520 con 0x7f8220007230 2024-03-29T11:00:32.138 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.156+0000 7f8232ffd640 1 -- 172.21.15.71:0/2556991170 >> v1:172.21.15.71:6790/0 conn(0x7f82200036e0 legacy=0x7f8220005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:32.138 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.156+0000 7f8232ffd640 1 -- 172.21.15.71:0/2556991170 >> v1:172.21.15.137:6789/0 conn(0x7f822000af00 legacy=0x7f822000d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:32.138 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.156+0000 7f8232ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:00:32.138 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.156+0000 7f8232ffd640 1 monclient: found mon.noname-a 2024-03-29T11:00:32.138 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.156+0000 7f8232ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:32.138 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.156+0000 7f8232ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:00:32.138 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.156+0000 7f8232ffd640 1 -- 172.21.15.71:0/2556991170 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8220012090 con 0x7f8220007230 2024-03-29T11:00:32.138 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.156+0000 7f8232ffd640 10 monclient: _finish_auth 0 2024-03-29T11:00:32.138 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.156+0000 7f8232ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:00:32.138 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.156+0000 7f8232ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:32.138 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.156+0000 7f8232ffd640 1 -- 172.21.15.71:0/2556991170 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f82280016d0 con 0x7f8220007230 2024-03-29T11:00:32.138 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.156+0000 7f8232ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:32.139 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.156+0000 7f8232ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:00:32.139 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.156+0000 7f8232ffd640 10 monclient: dump: 2024-03-29T11:00:32.139 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-29T11:00:32.139 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:32.139 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:32.139 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:32.139 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:32.139 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:32.139 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:32.139 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:32.139 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:32.139 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:00:32.139 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.156+0000 7f823a657640 5 monclient: authenticate success, global_id 4101 2024-03-29T11:00:32.139 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.156+0000 7f823a657640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:32.140 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.156+0000 7f8232ffd640 1 -- 172.21.15.71:0/2556991170 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82280018a0 con 0x7f8220007230 2024-03-29T11:00:32.140 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.156+0000 7f8232ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:32.140 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.156+0000 7f8232ffd640 1 -- 172.21.15.71:0/2556991170 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f8228001c30 con 0x7f8220007230 2024-03-29T11:00:32.140 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.156+0000 7f823a657640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:32.140 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.156+0000 7f823a657640 4 set_mon_vals no callback set 2024-03-29T11:00:32.140 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.156+0000 7f8232ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:32.140 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.156+0000 7f8232ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:32.140 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.156+0000 7f8232ffd640 10 monclient: dump: 2024-03-29T11:00:32.140 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-29T11:00:32.140 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:32.140 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:32.140 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:32.140 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:32.140 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:32.141 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:32.141 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:32.141 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:32.141 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:00:32.148 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.157+0000 7f823a657640 10 monclient: shutdown 2024-03-29T11:00:32.148 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.157+0000 7f823a657640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:32.148 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.157+0000 7f823a657640 1 -- 172.21.15.71:0/2556991170 >> v1:172.21.15.71:6789/0 conn(0x7f8220007230 legacy=0x7f82200096a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:32.148 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.157+0000 7f823a657640 1 -- 172.21.15.71:0/2556991170 shutdown_connections 2024-03-29T11:00:32.148 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.157+0000 7f823a657640 1 -- 172.21.15.71:0/2556991170 >> 172.21.15.71:0/2556991170 conn(0x7f8234066b00 msgr2=0x7f8234066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:32.148 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.157+0000 7f823a657640 1 -- 172.21.15.71:0/2556991170 shutdown_connections 2024-03-29T11:00:32.148 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.158+0000 7f823a657640 1 -- 172.21.15.71:0/2556991170 wait complete. 2024-03-29T11:00:32.148 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.158+0000 7f823a657640 10 monclient: build_initial_monmap 2024-03-29T11:00:32.148 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.158+0000 7f823a657640 10 monclient: monmap: 2024-03-29T11:00:32.148 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-03-29T11:00:32.148 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:32.149 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 0.000000 2024-03-29T11:00:32.149 INFO:teuthology.orchestra.run.smithi071.stderr:created 0.000000 2024-03-29T11:00:32.149 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:32.149 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:32.149 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:32.149 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:00:32.149 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:00:32.149 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:00:32.149 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.159+0000 7f823a657640 1 librados: starting msgr at 2024-03-29T11:00:32.149 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.159+0000 7f823a657640 1 librados: starting objecter 2024-03-29T11:00:32.149 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.159+0000 7f823a657640 1 Processor -- start 2024-03-29T11:00:32.149 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.159+0000 7f823a657640 1 -- start start 2024-03-29T11:00:32.149 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.159+0000 7f823a657640 1 librados: setting wanted keys 2024-03-29T11:00:32.149 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.159+0000 7f823a657640 1 librados: calling monclient init 2024-03-29T11:00:32.150 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.159+0000 7f823a657640 10 monclient: init 2024-03-29T11:00:32.150 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.160+0000 7f823a657640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:32.150 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.160+0000 7f823a657640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:00:32.150 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.160+0000 7f823a657640 10 monclient(hunting): picked mon.noname-c con 0x7f82200036e0 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:32.150 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.160+0000 7f823a657640 10 monclient(hunting): picked mon.noname-b con 0x7f8220007230 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:32.150 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.160+0000 7f823a657640 10 monclient(hunting): picked mon.noname-a con 0x7f822000af00 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:32.150 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.160+0000 7f823a657640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8234136440 con 0x7f822000af00 2024-03-29T11:00:32.150 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.160+0000 7f823a657640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8234135fb0 con 0x7f8220007230 2024-03-29T11:00:32.150 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.160+0000 7f823a657640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8234136140 con 0x7f82200036e0 2024-03-29T11:00:32.150 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.160+0000 7f823a657640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:32.150 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.160+0000 7f823a657640 10 monclient(hunting): authenticate will time out at 788.419641s 2024-03-29T11:00:32.150 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.160+0000 7f8233fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f82200036e0 0x7f82341358a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:38254/0 (socket says 172.21.15.71:38254) 2024-03-29T11:00:32.150 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.160+0000 7f8233fff640 1 -- 172.21.15.71:0/2843425227 learned_addr learned my addr 172.21.15.71:0/2843425227 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:00:32.151 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.160+0000 7f8230ff9640 10 client.?.objecter ms_handle_connect 0x7f82200036e0 2024-03-29T11:00:32.151 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.160+0000 7f8230ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:32.151 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.160+0000 7f82337fe640 1 -- 172.21.15.71:0/2843425227 >> v1:172.21.15.137:6789/0 conn(0x7f8220007230 legacy=0x7f82340b0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-29T11:00:32.151 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.160+0000 7f8230ff9640 10 client.?.objecter ms_handle_connect 0x7f822000af00 2024-03-29T11:00:32.151 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.160+0000 7f8230ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:32.151 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.160+0000 7f82337fe640 1 -- 172.21.15.71:0/2843425227 >> v1:172.21.15.137:6789/0 conn(0x7f8220007230 legacy=0x7f82340b0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:00:32.151 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.160+0000 7f82337fe640 1 --1- 172.21.15.71:0/2843425227 >> v1:172.21.15.137:6789/0 conn(0x7f8220007230 0x7f82340b0f70 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-03-29T11:00:32.151 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.160+0000 7f82337fe640 1 --1- 172.21.15.71:0/2843425227 >> v1:172.21.15.137:6789/0 conn(0x7f8220007230 0x7f82340b0f70 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-29T11:00:32.151 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.160+0000 7f8230ff9640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.137:6789/0 2024-03-29T11:00:32.151 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.161+0000 7f8230ff9640 1 -- 172.21.15.71:0/2843425227 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842206333 0 0) 0x7f8234136140 con 0x7f82200036e0 2024-03-29T11:00:32.151 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.161+0000 7f8230ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:00:32.151 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.161+0000 7f8230ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:32.151 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.161+0000 7f8230ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:32.151 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.161+0000 7f8230ff9640 10 monclient(hunting): my global_id is 4109 2024-03-29T11:00:32.152 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.161+0000 7f8230ff9640 1 -- 172.21.15.71:0/2843425227 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8220009990 con 0x7f82200036e0 2024-03-29T11:00:32.152 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.161+0000 7f8230ff9640 1 -- 172.21.15.71:0/2843425227 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1538919425 0 0) 0x7f8234136440 con 0x7f822000af00 2024-03-29T11:00:32.152 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.161+0000 7f8230ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:00:32.152 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.161+0000 7f8230ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:32.152 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.161+0000 7f8230ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:32.152 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.161+0000 7f8230ff9640 10 monclient(hunting): my global_id is 4110 2024-03-29T11:00:32.152 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.161+0000 7f8230ff9640 1 -- 172.21.15.71:0/2843425227 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8234136140 con 0x7f822000af00 2024-03-29T11:00:32.152 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.161+0000 7f8230ff9640 1 -- 172.21.15.71:0/2843425227 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2543396077 0 0) 0x7f8234136140 con 0x7f822000af00 2024-03-29T11:00:32.152 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.161+0000 7f8230ff9640 1 -- 172.21.15.71:0/2843425227 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8234136440 con 0x7f822000af00 2024-03-29T11:00:32.152 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.161+0000 7f8230ff9640 1 -- 172.21.15.71:0/2843425227 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f822c001a90 con 0x7f822000af00 2024-03-29T11:00:32.152 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.161+0000 7f8230ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:32.152 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.161+0000 7f8230ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:00:32.152 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.161+0000 7f8230ff9640 10 monclient(hunting): dump: 2024-03-29T11:00:32.152 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-29T11:00:32.152 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:32.153 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:32.153 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:32.153 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:32.153 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:32.153 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:32.153 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:32.153 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:32.153 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:00:32.153 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.161+0000 7f8230ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:32.153 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.162+0000 7f8230ff9640 1 -- 172.21.15.71:0/2843425227 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3592282047 0 0) 0x7f8234136440 con 0x7f822000af00 2024-03-29T11:00:32.153 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.162+0000 7f8230ff9640 1 -- 172.21.15.71:0/2843425227 >> v1:172.21.15.71:6790/0 conn(0x7f82200036e0 legacy=0x7f82341358a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:32.153 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.162+0000 7f8230ff9640 1 -- 172.21.15.71:0/2843425227 >> v1:172.21.15.137:6789/0 conn(0x7f8220007230 legacy=0x7f82340b0f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:00:32.153 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.162+0000 7f8230ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:00:32.153 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.162+0000 7f8230ff9640 1 monclient: found mon.a 2024-03-29T11:00:32.153 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.162+0000 7f8230ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:32.154 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.162+0000 7f8230ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:00:32.154 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.162+0000 7f8230ff9640 1 -- 172.21.15.71:0/2843425227 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8234134f50 con 0x7f822000af00 2024-03-29T11:00:32.154 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.162+0000 7f8230ff9640 10 monclient: _finish_auth 0 2024-03-29T11:00:32.154 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.162+0000 7f8230ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:00:32.154 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.162+0000 7f8230ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:32.154 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.162+0000 7f823a657640 5 monclient: authenticate success, global_id 4110 2024-03-29T11:00:32.154 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.162+0000 7f823a657640 10 monclient: _renew_subs 2024-03-29T11:00:32.154 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.162+0000 7f823a657640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:00:32.154 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.162+0000 7f823a657640 1 -- 172.21.15.71:0/2843425227 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82341351a0 con 0x7f822000af00 2024-03-29T11:00:32.154 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.162+0000 7f823a657640 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:32.154 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.162+0000 7f823a657640 10 monclient: _renew_subs 2024-03-29T11:00:32.154 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.162+0000 7f823a657640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:00:32.154 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.162+0000 7f823a657640 1 -- 172.21.15.71:0/2843425227 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82340b1860 con 0x7f822000af00 2024-03-29T11:00:32.155 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.162+0000 7f823a657640 1 librados: init done 2024-03-29T11:00:32.155 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.164+0000 7f823a657640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:32.155 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.164+0000 7f823a657640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:32.155 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.164+0000 7f823a657640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:00:32.155 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.164+0000 7f823a657640 1 -- 172.21.15.71:0/2843425227 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82341d6ca0 con 0x7f822000af00 2024-03-29T11:00:32.155 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.164+0000 7f8230ff9640 1 -- 172.21.15.71:0/2843425227 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f822c002020 con 0x7f822000af00 2024-03-29T11:00:32.155 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.164+0000 7f8230ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:32.155 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.164+0000 7f8230ff9640 1 -- 172.21.15.71:0/2843425227 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f822c003b10 con 0x7f822000af00 2024-03-29T11:00:32.155 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.164+0000 7f8230ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:32.155 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.164+0000 7f8230ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:32.155 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.164+0000 7f8230ff9640 10 monclient: dump: 2024-03-29T11:00:32.155 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-29T11:00:32.155 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:32.155 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:32.156 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:32.156 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:32.156 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:32.156 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:32.156 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:32.156 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:32.156 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:00:32.156 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.164+0000 7f8230ff9640 1 -- 172.21.15.71:0/2843425227 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f822c0040c0 con 0x7f822000af00 2024-03-29T11:00:32.156 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.164+0000 7f82327fc640 4 set_mon_vals no callback set 2024-03-29T11:00:32.156 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.164+0000 7f8230ff9640 1 -- 172.21.15.71:0/2843425227 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3223129556 0 0) 0x7f822c005530 con 0x7f822000af00 2024-03-29T11:00:32.156 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.164+0000 7f8230ff9640 10 client.4110.objecter ms_dispatch 0x7f8234000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:00:32.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.164+0000 7f8230ff9640 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:00:32.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.164+0000 7f8230ff9640 3 client.4110.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:00:32.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.164+0000 7f8230ff9640 20 client.4110.objecter dump_active .. 0 homeless 2024-03-29T11:00:32.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.166+0000 7f8230ff9640 1 -- 172.21.15.71:0/2843425227 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f822c004550 con 0x7f822000af00 2024-03-29T11:00:32.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.166+0000 7f8230ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:32.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.166+0000 7f8230ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:32.217 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.236+0000 7f823a657640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:00:32.218 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.236+0000 7f823a657640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:00:32.218 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.237+0000 7f823a657640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:00:32.218 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.237+0000 7f823a657640 1 -- 172.21.15.71:0/2843425227 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f823409b020 con 0x7f822000af00 2024-03-29T11:00:32.219 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.238+0000 7f8230ff9640 1 -- 172.21.15.71:0/2843425227 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f822c004790 con 0x7f822000af00 2024-03-29T11:00:32.219 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.238+0000 7f8230ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:00:32.219 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.238+0000 7f8230ff9640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:00:32.219 INFO:teuthology.orchestra.run.smithi071.stderr:adjusted tunables profile to default 2024-03-29T11:00:32.221 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.239+0000 7f823a657640 10 librados: watch_flush enter 2024-03-29T11:00:32.221 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.239+0000 7f823a657640 10 librados: watch_flush exit 2024-03-29T11:00:32.221 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.239+0000 7f823a657640 20 client.4110.objecter shutdown clearing up homeless session... 2024-03-29T11:00:32.221 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.239+0000 7f823a657640 10 client.4110.objecter successfully canceled tick 2024-03-29T11:00:32.221 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.239+0000 7f823a657640 10 monclient: shutdown 2024-03-29T11:00:32.221 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.239+0000 7f823a657640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:32.221 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.239+0000 7f823a657640 1 -- 172.21.15.71:0/2843425227 >> v1:172.21.15.71:6789/0 conn(0x7f822000af00 legacy=0x7f82341d6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:32.221 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.239+0000 7f823a657640 1 -- 172.21.15.71:0/2843425227 shutdown_connections 2024-03-29T11:00:32.221 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.239+0000 7f823a657640 1 -- 172.21.15.71:0/2843425227 >> 172.21.15.71:0/2843425227 conn(0x7f8234066b00 msgr2=0x7f823413bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:32.222 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.240+0000 7f823a657640 1 -- 172.21.15.71:0/2843425227 shutdown_connections 2024-03-29T11:00:32.222 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.240+0000 7f823a657640 1 -- 172.21.15.71:0/2843425227 wait complete. 2024-03-29T11:00:32.222 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:32.240+0000 7f823a657640 1 librados: shutdown 2024-03-29T11:00:32.235 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:00:32.235 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:00:32.235 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:00:32.235 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:00:32.262 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:00:32.262 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:00:32.330 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:00:32.330 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:00:32.396 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:00:32.396 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:00:32.463 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T11:00:32.463 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:00:32.490 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T11:00:32.490 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:00:32.558 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T11:00:32.558 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:00:32.623 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T11:00:32.643+0000 7fab49e36040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:00:32.625 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-29T11:00:32.625 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:00:32.646 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:00:32.665+0000 7f5976230040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:00:32.679 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T11:00:32.699+0000 7fab49e36040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:00:32.692 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new 6ae3879f-03d5-4971-92a9-8a7dce1e877d 0 2024-03-29T11:00:32.700 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:00:32.719+0000 7f5976230040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:00:32.832 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.852+0000 7fc8a580d640 10 monclient: get_monmap_and_config 2024-03-29T11:00:32.832 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.852+0000 7fc8a580d640 10 monclient: build_initial_monmap 2024-03-29T11:00:32.832 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.852+0000 7fc8a580d640 10 monclient: monmap: 2024-03-29T11:00:32.832 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 0 2024-03-29T11:00:32.832 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:32.832 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T11:00:32.853290+0000 2024-03-29T11:00:32.832 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T11:00:32.853290+0000 2024-03-29T11:00:32.832 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:32.832 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:32.832 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:32.833 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:00:32.833 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:00:32.833 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:32.842 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.852+0000 7fc8a580d640 1 Processor -- start 2024-03-29T11:00:32.842 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.852+0000 7fc8a580d640 1 -- start start 2024-03-29T11:00:32.843 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.852+0000 7fc8a580d640 10 monclient: init 2024-03-29T11:00:32.843 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc8a580d640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:32.843 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc8a580d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:00:32.843 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc8a580d640 10 monclient(hunting): picked mon.noname-b con 0x7fc8a009ac40 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:32.843 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc8a580d640 10 monclient(hunting): picked mon.noname-a con 0x7fc8a009c870 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:32.843 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc8a580d640 10 monclient(hunting): picked mon.noname-c con 0x7fc8a00b4360 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:32.843 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc8a580d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a009dec0 con 0x7fc8a009c870 2024-03-29T11:00:32.843 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc8a580d640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a009da30 con 0x7fc8a00b4360 2024-03-29T11:00:32.843 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc8a580d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a00b39c0 con 0x7fc8a009ac40 2024-03-29T11:00:32.843 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc8a580d640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:32.843 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc8a580d640 10 monclient(hunting): authenticate will time out at 778.187455s 2024-03-29T11:00:32.843 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc89f7fe640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fc8a00b4360 0x7fc8a014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.137:46838/0 (socket says 172.21.15.137:46838) 2024-03-29T11:00:32.843 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc89f7fe640 1 -- 172.21.15.137:0/449501416 learned_addr learned my addr 172.21.15.137:0/449501416 (peer_addr_for_me v1:172.21.15.137:0/0) 2024-03-29T11:00:32.843 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc89dffb640 1 -- 172.21.15.137:0/449501416 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1962143425 0 0) 0x7fc8a009da30 con 0x7fc8a00b4360 2024-03-29T11:00:32.844 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc89dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:32.844 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc89dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:32.844 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc89dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:32.844 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc89dffb640 10 monclient(hunting): my global_id is 4108 2024-03-29T11:00:32.844 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc89dffb640 1 -- 172.21.15.137:0/449501416 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc888003520 con 0x7fc8a00b4360 2024-03-29T11:00:32.844 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc89dffb640 1 -- 172.21.15.137:0/449501416 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1041256632 0 0) 0x7fc890001110 con 0x7fc8a009c870 2024-03-29T11:00:32.844 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc89dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:32.844 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc89dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:32.844 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc89dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:32.844 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc89dffb640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:00:32.844 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc89dffb640 1 -- 172.21.15.137:0/449501416 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8a009da30 con 0x7fc8a009c870 2024-03-29T11:00:32.844 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc89dffb640 1 -- 172.21.15.137:0/449501416 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2766847394 0 0) 0x7fc8a00b39c0 con 0x7fc8a009ac40 2024-03-29T11:00:32.844 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc89dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:32.844 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc89dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:32.844 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc89dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:32.845 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc89dffb640 10 monclient(hunting): my global_id is 4112 2024-03-29T11:00:32.845 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.853+0000 7fc89dffb640 1 -- 172.21.15.137:0/449501416 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc890001110 con 0x7fc8a009ac40 2024-03-29T11:00:32.845 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.854+0000 7fc89dffb640 1 -- 172.21.15.137:0/449501416 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2542326659 0 0) 0x7fc888003520 con 0x7fc8a00b4360 2024-03-29T11:00:32.845 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.854+0000 7fc89dffb640 1 -- 172.21.15.137:0/449501416 >> v1:172.21.15.71:6790/0 conn(0x7fc8a009ac40 legacy=0x7fc8a00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:32.845 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.854+0000 7fc89dffb640 1 -- 172.21.15.137:0/449501416 >> v1:172.21.15.71:6789/0 conn(0x7fc8a009c870 legacy=0x7fc8a013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:32.845 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.854+0000 7fc89dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:00:32.845 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.854+0000 7fc89dffb640 1 monclient: found mon.noname-c 2024-03-29T11:00:32.845 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.854+0000 7fc89dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:32.845 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.854+0000 7fc89dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.137:6789/0 2024-03-29T11:00:32.845 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.854+0000 7fc89dffb640 1 -- 172.21.15.137:0/449501416 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a00b5160 con 0x7fc8a00b4360 2024-03-29T11:00:32.846 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.854+0000 7fc89dffb640 10 monclient: _finish_auth 0 2024-03-29T11:00:32.846 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.854+0000 7fc89dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:00:32.846 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.854+0000 7fc89dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:32.846 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.854+0000 7fc89dffb640 1 -- 172.21.15.137:0/449501416 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fc898001a10 con 0x7fc8a00b4360 2024-03-29T11:00:32.846 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.854+0000 7fc8a580d640 5 monclient: authenticate success, global_id 4108 2024-03-29T11:00:32.846 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.854+0000 7fc8a580d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:32.846 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.854+0000 7fc89dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:32.846 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.854+0000 7fc89dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:32.846 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.854+0000 7fc89dffb640 10 monclient: dump: 2024-03-29T11:00:32.846 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:32.846 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:32.846 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:32.846 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:32.846 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:32.846 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:32.847 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:32.847 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:32.847 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:32.847 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:32.847 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.854+0000 7fc8a580d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:32.847 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.854+0000 7fc89dffb640 1 -- 172.21.15.137:0/449501416 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc898001c60 con 0x7fc8a00b4360 2024-03-29T11:00:32.847 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.854+0000 7fc89dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:32.847 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.854+0000 7fc89dffb640 1 -- 172.21.15.137:0/449501416 <== mon.1 v1:172.21.15.137:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fc898001ff0 con 0x7fc8a00b4360 2024-03-29T11:00:32.847 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.854+0000 7fc89dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:32.847 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.854+0000 7fc89dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:32.848 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.854+0000 7fc89dffb640 10 monclient: dump: 2024-03-29T11:00:32.848 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:32.848 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:32.848 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:32.848 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:32.848 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:32.848 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:32.848 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:32.848 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:32.848 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:32.848 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:32.848 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.854+0000 7fc8a580d640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:32.848 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.854+0000 7fc8a580d640 4 set_mon_vals no callback set 2024-03-29T11:00:32.848 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.855+0000 7fc8a580d640 10 monclient: shutdown 2024-03-29T11:00:32.848 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.855+0000 7fc8a580d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:32.849 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.855+0000 7fc8a580d640 1 -- 172.21.15.137:0/449501416 >> v1:172.21.15.137:6789/0 conn(0x7fc8a00b4360 legacy=0x7fc8a014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:32.849 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.855+0000 7fc8a580d640 1 -- 172.21.15.137:0/449501416 shutdown_connections 2024-03-29T11:00:32.849 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.855+0000 7fc8a580d640 1 -- 172.21.15.137:0/449501416 >> 172.21.15.137:0/449501416 conn(0x7fc8a0066b00 msgr2=0x7fc8a0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:32.849 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.855+0000 7fc8a580d640 1 -- 172.21.15.137:0/449501416 shutdown_connections 2024-03-29T11:00:32.849 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.855+0000 7fc8a580d640 1 -- 172.21.15.137:0/449501416 wait complete. 2024-03-29T11:00:32.849 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.855+0000 7fc8a580d640 10 monclient: build_initial_monmap 2024-03-29T11:00:32.849 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.855+0000 7fc8a580d640 10 monclient: monmap: 2024-03-29T11:00:32.849 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 0 2024-03-29T11:00:32.849 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:32.849 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 0.000000 2024-03-29T11:00:32.849 INFO:teuthology.orchestra.run.smithi137.stderr:created 0.000000 2024-03-29T11:00:32.849 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:32.850 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:32.850 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:32.850 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:00:32.850 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:00:32.850 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:32.850 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.856+0000 7fc8a580d640 1 librados: starting msgr at 2024-03-29T11:00:32.850 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.856+0000 7fc8a580d640 1 librados: starting objecter 2024-03-29T11:00:32.850 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.856+0000 7fc8a580d640 1 Processor -- start 2024-03-29T11:00:32.850 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.856+0000 7fc8a580d640 1 -- start start 2024-03-29T11:00:32.850 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.856+0000 7fc8a580d640 1 librados: setting wanted keys 2024-03-29T11:00:32.850 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.856+0000 7fc8a580d640 1 librados: calling monclient init 2024-03-29T11:00:32.850 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.856+0000 7fc8a580d640 10 monclient: init 2024-03-29T11:00:32.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.857+0000 7fc8a580d640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:32.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.857+0000 7fc8a580d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:00:32.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.857+0000 7fc8a580d640 10 monclient(hunting): picked mon.noname-b con 0x7fc8a009ac40 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:32.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.857+0000 7fc8a580d640 10 monclient(hunting): picked mon.noname-c con 0x7fc8a01415f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:32.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.857+0000 7fc8a580d640 10 monclient(hunting): picked mon.noname-a con 0x7fc8a01d7d50 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:32.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.857+0000 7fc8a580d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a0141440 con 0x7fc8a01d7d50 2024-03-29T11:00:32.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.857+0000 7fc8a580d640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a0141ca0 con 0x7fc8a009ac40 2024-03-29T11:00:32.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.857+0000 7fc8a580d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a0142e40 con 0x7fc8a01415f0 2024-03-29T11:00:32.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.857+0000 7fc8a580d640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:32.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.857+0000 7fc8a580d640 10 monclient(hunting): authenticate will time out at 778.191750s 2024-03-29T11:00:32.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.857+0000 7fc89effd640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fc8a009ac40 0x7fc8a0140d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.137:46844/0 (socket says 172.21.15.137:46844) 2024-03-29T11:00:32.852 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.857+0000 7fc89effd640 1 -- 172.21.15.137:0/1616181205 learned_addr learned my addr 172.21.15.137:0/1616181205 (peer_addr_for_me v1:172.21.15.137:0/0) 2024-03-29T11:00:32.852 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.857+0000 7fc887fff640 10 client.?.objecter ms_handle_connect 0x7fc8a009ac40 2024-03-29T11:00:32.852 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.857+0000 7fc887fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:32.852 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.857+0000 7fc887fff640 10 client.?.objecter ms_handle_connect 0x7fc8a01415f0 2024-03-29T11:00:32.852 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.857+0000 7fc887fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:32.852 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.857+0000 7fc887fff640 1 -- 172.21.15.137:0/1616181205 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4276646604 0 0) 0x7fc8a0141ca0 con 0x7fc8a009ac40 2024-03-29T11:00:32.852 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:32.852 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:32.852 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:32.852 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient(hunting): my global_id is 4111 2024-03-29T11:00:32.852 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 1 -- 172.21.15.137:0/1616181205 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc88c004eb0 con 0x7fc8a009ac40 2024-03-29T11:00:32.852 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 client.?.objecter ms_handle_connect 0x7fc8a01d7d50 2024-03-29T11:00:32.853 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:32.853 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 1 -- 172.21.15.137:0/1616181205 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3138173027 0 0) 0x7fc8a0141440 con 0x7fc8a01d7d50 2024-03-29T11:00:32.853 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:32.853 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:32.853 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:32.853 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:00:32.853 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 1 -- 172.21.15.137:0/1616181205 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8a0141ca0 con 0x7fc8a01d7d50 2024-03-29T11:00:32.853 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 1 -- 172.21.15.137:0/1616181205 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257270690 0 0) 0x7fc8a0142e40 con 0x7fc8a01415f0 2024-03-29T11:00:32.853 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:32.853 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:32.853 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:32.853 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:00:32.854 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 1 -- 172.21.15.137:0/1616181205 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8a0141440 con 0x7fc8a01415f0 2024-03-29T11:00:32.854 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 1 -- 172.21.15.137:0/1616181205 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3413978533 0 0) 0x7fc8a0141ca0 con 0x7fc8a01d7d50 2024-03-29T11:00:32.854 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 1 -- 172.21.15.137:0/1616181205 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8a0142e40 con 0x7fc8a01d7d50 2024-03-29T11:00:32.854 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 1 -- 172.21.15.137:0/1616181205 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fc898001f70 con 0x7fc8a01d7d50 2024-03-29T11:00:32.854 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:32.854 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:00:32.854 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient(hunting): dump: 2024-03-29T11:00:32.854 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:32.854 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:32.854 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:32.854 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:32.855 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:32.855 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:32.855 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:32.855 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:32.855 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:32.855 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:32.855 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:32.855 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 1 -- 172.21.15.137:0/1616181205 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1782467435 0 0) 0x7fc88c004eb0 con 0x7fc8a009ac40 2024-03-29T11:00:32.855 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 1 -- 172.21.15.137:0/1616181205 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8a0141ca0 con 0x7fc8a009ac40 2024-03-29T11:00:32.855 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 1 -- 172.21.15.137:0/1616181205 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 926201014 0 0) 0x7fc8a0141440 con 0x7fc8a01415f0 2024-03-29T11:00:32.855 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 1 -- 172.21.15.137:0/1616181205 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc88c004eb0 con 0x7fc8a01415f0 2024-03-29T11:00:32.855 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 1 -- 172.21.15.137:0/1616181205 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fc894001930 con 0x7fc8a009ac40 2024-03-29T11:00:32.856 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:32.856 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:32.856 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient(hunting): dump: 2024-03-29T11:00:32.856 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:32.856 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:32.856 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:32.856 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:32.856 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:32.856 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:32.856 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:32.856 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:32.857 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:32.857 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:32.857 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 1 -- 172.21.15.137:0/1616181205 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fc8900018a0 con 0x7fc8a01415f0 2024-03-29T11:00:32.857 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:32.857 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:32.857 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient(hunting): dump: 2024-03-29T11:00:32.857 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:32.857 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:32.857 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:32.857 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:32.858 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:32.858 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:32.858 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:32.858 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:32.858 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:32.858 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:32.858 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 1 -- 172.21.15.137:0/1616181205 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4285877640 0 0) 0x7fc8a0142e40 con 0x7fc8a01d7d50 2024-03-29T11:00:32.858 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 1 -- 172.21.15.137:0/1616181205 >> v1:172.21.15.71:6790/0 conn(0x7fc8a01415f0 legacy=0x7fc8a01d7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:32.858 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 1 -- 172.21.15.137:0/1616181205 >> v1:172.21.15.137:6789/0 conn(0x7fc8a009ac40 legacy=0x7fc8a0140d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:32.858 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient: _finish_hunting 0 2024-03-29T11:00:32.858 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 1 monclient: found mon.a 2024-03-29T11:00:32.858 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:32.858 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:00:32.859 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 1 -- 172.21.15.137:0/1616181205 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a01d8180 con 0x7fc8a01d7d50 2024-03-29T11:00:32.859 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient: _finish_auth 0 2024-03-29T11:00:32.859 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient: _check_auth_tickets 2024-03-29T11:00:32.859 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:32.859 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc8a580d640 5 monclient: authenticate success, global_id 4116 2024-03-29T11:00:32.859 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc8a580d640 10 monclient: _renew_subs 2024-03-29T11:00:32.859 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc8a580d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:00:32.859 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc8a580d640 1 -- 172.21.15.137:0/1616181205 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8a01e3810 con 0x7fc8a01d7d50 2024-03-29T11:00:32.859 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc8a580d640 10 client.4116.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:32.859 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc8a580d640 10 monclient: _renew_subs 2024-03-29T11:00:32.859 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc8a580d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:00:32.859 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc8a580d640 1 -- 172.21.15.137:0/1616181205 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8a01e3b80 con 0x7fc8a01d7d50 2024-03-29T11:00:32.859 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc8a580d640 1 librados: init done 2024-03-29T11:00:32.859 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 1 -- 172.21.15.137:0/1616181205 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc898001420 con 0x7fc8a01d7d50 2024-03-29T11:00:32.859 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:32.860 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 1 -- 172.21.15.137:0/1616181205 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fc898004fb0 con 0x7fc8a01d7d50 2024-03-29T11:00:32.860 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:32.860 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc89d7fa640 4 set_mon_vals no callback set 2024-03-29T11:00:32.860 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:32.860 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 monclient: dump: 2024-03-29T11:00:32.860 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:32.860 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:32.860 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:32.860 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:32.860 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:32.860 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:32.860 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:32.860 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:32.860 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:32.861 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:32.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 1 -- 172.21.15.137:0/1616181205 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc8980054a0 con 0x7fc8a01d7d50 2024-03-29T11:00:32.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 1 -- 172.21.15.137:0/1616181205 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 2713274946 0 0) 0x7fc8980067c0 con 0x7fc8a01d7d50 2024-03-29T11:00:32.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 10 client.4116.objecter ms_dispatch 0x7fc8a0000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:00:32.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 3 client.4116.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:00:32.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 3 client.4116.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:00:32.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.858+0000 7fc887fff640 20 client.4116.objecter dump_active .. 0 homeless 2024-03-29T11:00:32.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.859+0000 7fc8a580d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:32.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.859+0000 7fc8a580d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:32.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.859+0000 7fc8a580d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:00:32.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.859+0000 7fc8a580d640 1 -- 172.21.15.137:0/1616181205 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc880005080 con 0x7fc8a01d7d50 2024-03-29T11:00:32.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.861+0000 7fc887fff640 1 -- 172.21.15.137:0/1616181205 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc898006a00 con 0x7fc8a01d7d50 2024-03-29T11:00:32.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.861+0000 7fc887fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:32.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.861+0000 7fc887fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:32.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.934+0000 7fc8a580d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6ae3879f-03d5-4971-92a9-8a7dce1e877d", "id": 0}] 2024-03-29T11:00:32.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.934+0000 7fc8a580d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6ae3879f-03d5-4971-92a9-8a7dce1e877d", "id": 0}] 2024-03-29T11:00:32.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.934+0000 7fc8a580d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:00:32.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.934+0000 7fc8a580d640 1 -- 172.21.15.137:0/1616181205 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd new", "uuid": "6ae3879f-03d5-4971-92a9-8a7dce1e877d", "id": 0} v 0) v1 -- 0x7fc8800058b0 con 0x7fc8a01d7d50 2024-03-29T11:00:32.915 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.935+0000 7fc887fff640 1 -- 172.21.15.137:0/1616181205 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6ae3879f-03d5-4971-92a9-8a7dce1e877d", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 2619659200 0 732707515) 0x7fc8980058d0 con 0x7fc8a01d7d50 2024-03-29T11:00:32.915 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.935+0000 7fc887fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6ae3879f-03d5-4971-92a9-8a7dce1e877d", "id": 0}] 2024-03-29T11:00:32.915 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.935+0000 7fc887fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:32.916 INFO:teuthology.orchestra.run.smithi137.stdout:0 2024-03-29T11:00:32.917 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.936+0000 7fc8a580d640 10 librados: watch_flush enter 2024-03-29T11:00:32.917 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.936+0000 7fc8a580d640 10 librados: watch_flush exit 2024-03-29T11:00:32.917 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.936+0000 7fc8a580d640 20 client.4116.objecter shutdown clearing up homeless session... 2024-03-29T11:00:32.917 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.936+0000 7fc8a580d640 10 client.4116.objecter successfully canceled tick 2024-03-29T11:00:32.917 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.936+0000 7fc8a580d640 10 monclient: shutdown 2024-03-29T11:00:32.917 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.936+0000 7fc8a580d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:32.917 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.936+0000 7fc8a580d640 1 -- 172.21.15.137:0/1616181205 >> v1:172.21.15.71:6789/0 conn(0x7fc8a01d7d50 legacy=0x7fc8a01e20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:32.917 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.936+0000 7fc8a580d640 1 -- 172.21.15.137:0/1616181205 shutdown_connections 2024-03-29T11:00:32.917 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.936+0000 7fc8a580d640 1 -- 172.21.15.137:0/1616181205 >> 172.21.15.137:0/1616181205 conn(0x7fc8a0066b00 msgr2=0x7fc8a013d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:32.917 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.936+0000 7fc8a580d640 1 -- 172.21.15.137:0/1616181205 shutdown_connections 2024-03-29T11:00:32.917 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.936+0000 7fc8a580d640 1 -- 172.21.15.137:0/1616181205 wait complete. 2024-03-29T11:00:32.917 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:32.936+0000 7fc8a580d640 1 librados: shutdown 2024-03-29T11:00:32.927 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new 5c6efb84-6592-41d7-b4f1-b4a33d2d2f83 1 2024-03-29T11:00:33.026 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.046+0000 7f5b15c1e640 10 monclient: get_monmap_and_config 2024-03-29T11:00:33.026 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.046+0000 7f5b15c1e640 10 monclient: build_initial_monmap 2024-03-29T11:00:33.026 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.046+0000 7f5b15c1e640 10 monclient: monmap: 2024-03-29T11:00:33.026 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 0 2024-03-29T11:00:33.026 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.027 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T11:00:33.047426+0000 2024-03-29T11:00:33.027 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T11:00:33.047426+0000 2024-03-29T11:00:33.027 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:33.027 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.027 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:33.027 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:00:33.027 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:00:33.027 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.036 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.047+0000 7f5b15c1e640 1 Processor -- start 2024-03-29T11:00:33.036 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.047+0000 7f5b15c1e640 1 -- start start 2024-03-29T11:00:33.036 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.047+0000 7f5b15c1e640 10 monclient: init 2024-03-29T11:00:33.036 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.047+0000 7f5b15c1e640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:33.036 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.047+0000 7f5b15c1e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:00:33.037 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.047+0000 7f5b15c1e640 10 monclient(hunting): picked mon.noname-c con 0x7f5b1009ac40 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:33.037 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.047+0000 7f5b15c1e640 10 monclient(hunting): picked mon.noname-b con 0x7f5b1009c870 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:33.037 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.047+0000 7f5b15c1e640 10 monclient(hunting): picked mon.noname-a con 0x7f5b100b4360 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:33.037 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.047+0000 7f5b15c1e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1009dec0 con 0x7f5b100b4360 2024-03-29T11:00:33.037 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.047+0000 7f5b15c1e640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1009da30 con 0x7f5b1009ac40 2024-03-29T11:00:33.037 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.047+0000 7f5b15c1e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b100b39c0 con 0x7f5b1009c870 2024-03-29T11:00:33.037 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.047+0000 7f5b15c1e640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:33.037 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.047+0000 7f5b15c1e640 10 monclient(hunting): authenticate will time out at 778.381750s 2024-03-29T11:00:33.037 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.047+0000 7f5b0f7fe640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f5b1009ac40 0x7f5b100b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.137:46852/0 (socket says 172.21.15.137:46852) 2024-03-29T11:00:33.037 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.047+0000 7f5b0f7fe640 1 -- 172.21.15.137:0/4282630773 learned_addr learned my addr 172.21.15.137:0/4282630773 (peer_addr_for_me v1:172.21.15.137:0/0) 2024-03-29T11:00:33.037 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 1 -- 172.21.15.137:0/4282630773 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 587953131 0 0) 0x7f5b1009da30 con 0x7f5b1009ac40 2024-03-29T11:00:33.037 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.037 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.037 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.038 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:00:33.038 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 1 -- 172.21.15.137:0/4282630773 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5af8003520 con 0x7f5b1009ac40 2024-03-29T11:00:33.038 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 1 -- 172.21.15.137:0/4282630773 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2109846461 0 0) 0x7f5b08001490 con 0x7f5b100b4360 2024-03-29T11:00:33.038 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.038 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.038 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.038 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:00:33.038 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 1 -- 172.21.15.137:0/4282630773 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b1009da30 con 0x7f5b100b4360 2024-03-29T11:00:33.038 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 1 -- 172.21.15.137:0/4282630773 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3219297038 0 0) 0x7f5b100b39c0 con 0x7f5b1009c870 2024-03-29T11:00:33.038 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.038 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.038 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.038 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:00:33.038 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 1 -- 172.21.15.137:0/4282630773 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b08001490 con 0x7f5b1009c870 2024-03-29T11:00:33.039 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 1 -- 172.21.15.137:0/4282630773 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1295460336 0 0) 0x7f5af8003520 con 0x7f5b1009ac40 2024-03-29T11:00:33.039 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 1 -- 172.21.15.137:0/4282630773 >> v1:172.21.15.71:6790/0 conn(0x7f5b1009c870 legacy=0x7f5b1013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.039 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 1 -- 172.21.15.137:0/4282630773 >> v1:172.21.15.71:6789/0 conn(0x7f5b100b4360 legacy=0x7f5b1014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.039 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:00:33.039 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 1 monclient: found mon.noname-c 2024-03-29T11:00:33.039 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:33.039 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.137:6789/0 2024-03-29T11:00:33.039 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 1 -- 172.21.15.137:0/4282630773 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b100b5160 con 0x7f5b1009ac40 2024-03-29T11:00:33.039 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:00:33.039 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:00:33.039 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:33.039 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 1 -- 172.21.15.137:0/4282630773 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f5b00001690 con 0x7f5b1009ac40 2024-03-29T11:00:33.039 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b15c1e640 5 monclient: authenticate success, global_id 4114 2024-03-29T11:00:33.039 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.039 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:33.040 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 10 monclient: dump: 2024-03-29T11:00:33.040 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.040 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.040 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.040 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.040 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.040 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.040 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.040 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.040 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.040 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.040 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b15c1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:33.040 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 1 -- 172.21.15.137:0/4282630773 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b00001870 con 0x7f5b1009ac40 2024-03-29T11:00:33.040 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:33.041 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 1 -- 172.21.15.137:0/4282630773 <== mon.1 v1:172.21.15.137:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f5b00001c80 con 0x7f5b1009ac40 2024-03-29T11:00:33.041 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b15c1e640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:33.041 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b15c1e640 4 set_mon_vals no callback set 2024-03-29T11:00:33.041 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.041 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:33.041 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b0e7fc640 10 monclient: dump: 2024-03-29T11:00:33.041 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.041 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.041 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.041 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.042 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.042 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.042 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.042 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.042 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.042 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.042 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b15c1e640 10 monclient: shutdown 2024-03-29T11:00:33.042 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b15c1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:33.042 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.048+0000 7f5b15c1e640 1 -- 172.21.15.137:0/4282630773 >> v1:172.21.15.137:6789/0 conn(0x7f5b1009ac40 legacy=0x7f5b100b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.042 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.049+0000 7f5b15c1e640 1 -- 172.21.15.137:0/4282630773 shutdown_connections 2024-03-29T11:00:33.042 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.049+0000 7f5b15c1e640 1 -- 172.21.15.137:0/4282630773 >> 172.21.15.137:0/4282630773 conn(0x7f5b10066b00 msgr2=0x7f5b10066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:33.042 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.049+0000 7f5b15c1e640 1 -- 172.21.15.137:0/4282630773 shutdown_connections 2024-03-29T11:00:33.042 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.049+0000 7f5b15c1e640 1 -- 172.21.15.137:0/4282630773 wait complete. 2024-03-29T11:00:33.043 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.049+0000 7f5b15c1e640 10 monclient: build_initial_monmap 2024-03-29T11:00:33.043 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.049+0000 7f5b15c1e640 10 monclient: monmap: 2024-03-29T11:00:33.043 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 0 2024-03-29T11:00:33.043 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:33.043 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 0.000000 2024-03-29T11:00:33.043 INFO:teuthology.orchestra.run.smithi137.stderr:created 0.000000 2024-03-29T11:00:33.043 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:33.043 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.043 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:33.043 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:00:33.043 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:00:33.043 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.043 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.049+0000 7f5b15c1e640 1 librados: starting msgr at 2024-03-29T11:00:33.043 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.049+0000 7f5b15c1e640 1 librados: starting objecter 2024-03-29T11:00:33.043 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.050+0000 7f5b15c1e640 1 Processor -- start 2024-03-29T11:00:33.044 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.050+0000 7f5b15c1e640 1 -- start start 2024-03-29T11:00:33.044 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.050+0000 7f5b15c1e640 1 librados: setting wanted keys 2024-03-29T11:00:33.044 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.050+0000 7f5b15c1e640 1 librados: calling monclient init 2024-03-29T11:00:33.044 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.050+0000 7f5b15c1e640 10 monclient: init 2024-03-29T11:00:33.044 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.050+0000 7f5b15c1e640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:33.044 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.050+0000 7f5b15c1e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:00:33.044 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.050+0000 7f5b15c1e640 10 monclient(hunting): picked mon.noname-b con 0x7f5b1009ac40 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:33.044 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.050+0000 7f5b15c1e640 10 monclient(hunting): picked mon.noname-a con 0x7f5b100b12b0 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:33.044 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.050+0000 7f5b15c1e640 10 monclient(hunting): picked mon.noname-c con 0x7f5b101d7a40 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:33.044 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.050+0000 7f5b15c1e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b100b8000 con 0x7f5b100b12b0 2024-03-29T11:00:33.044 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.050+0000 7f5b15c1e640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b10142160 con 0x7f5b1009ac40 2024-03-29T11:00:33.044 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.050+0000 7f5b15c1e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b10142f70 con 0x7f5b101d7a40 2024-03-29T11:00:33.044 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.050+0000 7f5b15c1e640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:33.044 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.050+0000 7f5b15c1e640 10 monclient(hunting): authenticate will time out at 778.384799s 2024-03-29T11:00:33.045 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.050+0000 7f5b0f7fe640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f5b1009ac40 0x7f5b10142860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.137:46856/0 (socket says 172.21.15.137:46856) 2024-03-29T11:00:33.045 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.050+0000 7f5b0f7fe640 1 -- 172.21.15.137:0/565383034 learned_addr learned my addr 172.21.15.137:0/565383034 (peer_addr_for_me v1:172.21.15.137:0/0) 2024-03-29T11:00:33.045 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.050+0000 7f5af7fff640 10 client.?.objecter ms_handle_connect 0x7f5b1009ac40 2024-03-29T11:00:33.045 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.050+0000 7f5af7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:33.045 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 1 -- 172.21.15.137:0/565383034 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1250771467 0 0) 0x7f5b10142160 con 0x7f5b1009ac40 2024-03-29T11:00:33.045 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.045 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.045 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.045 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:00:33.045 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 1 -- 172.21.15.137:0/565383034 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5afc004eb0 con 0x7f5b1009ac40 2024-03-29T11:00:33.045 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 client.?.objecter ms_handle_connect 0x7f5b101d7a40 2024-03-29T11:00:33.045 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:33.045 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 client.?.objecter ms_handle_connect 0x7f5b100b12b0 2024-03-29T11:00:33.045 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:33.046 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 1 -- 172.21.15.137:0/565383034 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 98068406 0 0) 0x7f5b100b8000 con 0x7f5b100b12b0 2024-03-29T11:00:33.046 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.046 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.046 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.046 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:00:33.046 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 1 -- 172.21.15.137:0/565383034 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b10142160 con 0x7f5b100b12b0 2024-03-29T11:00:33.046 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 1 -- 172.21.15.137:0/565383034 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2438998261 0 0) 0x7f5b10142f70 con 0x7f5b101d7a40 2024-03-29T11:00:33.046 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.046 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.046 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.046 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:00:33.046 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 1 -- 172.21.15.137:0/565383034 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b100b8000 con 0x7f5b101d7a40 2024-03-29T11:00:33.046 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 1 -- 172.21.15.137:0/565383034 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3225180080 0 0) 0x7f5afc004eb0 con 0x7f5b1009ac40 2024-03-29T11:00:33.047 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 1 -- 172.21.15.137:0/565383034 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b10142f70 con 0x7f5b1009ac40 2024-03-29T11:00:33.047 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 1 -- 172.21.15.137:0/565383034 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f5b00001c40 con 0x7f5b1009ac40 2024-03-29T11:00:33.047 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.047 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:33.047 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient(hunting): dump: 2024-03-29T11:00:33.047 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.047 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.047 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.047 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.047 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.047 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.047 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.047 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.047 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.047 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.048 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:33.048 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 1 -- 172.21.15.137:0/565383034 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1466152112 0 0) 0x7f5b10142160 con 0x7f5b100b12b0 2024-03-29T11:00:33.048 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 1 -- 172.21.15.137:0/565383034 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5afc004eb0 con 0x7f5b100b12b0 2024-03-29T11:00:33.048 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 1 -- 172.21.15.137:0/565383034 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f5b040017a0 con 0x7f5b100b12b0 2024-03-29T11:00:33.048 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.048 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:33.048 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient(hunting): dump: 2024-03-29T11:00:33.048 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.048 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.048 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.048 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.048 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.048 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.048 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.048 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.049 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.049 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.049 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 1 -- 172.21.15.137:0/565383034 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 542180604 0 0) 0x7f5b100b8000 con 0x7f5b101d7a40 2024-03-29T11:00:33.049 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 1 -- 172.21.15.137:0/565383034 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b10142160 con 0x7f5b101d7a40 2024-03-29T11:00:33.049 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 1 -- 172.21.15.137:0/565383034 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f5b08001a00 con 0x7f5b101d7a40 2024-03-29T11:00:33.049 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.049 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:33.049 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient(hunting): dump: 2024-03-29T11:00:33.049 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.049 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.049 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.049 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.049 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.049 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.050 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.050 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.050 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.050 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.050 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 1 -- 172.21.15.137:0/565383034 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2457778188 0 0) 0x7f5b10142f70 con 0x7f5b1009ac40 2024-03-29T11:00:33.050 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 1 -- 172.21.15.137:0/565383034 >> v1:172.21.15.71:6790/0 conn(0x7f5b101d7a40 legacy=0x7f5b101e1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.050 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 1 -- 172.21.15.137:0/565383034 >> v1:172.21.15.71:6789/0 conn(0x7f5b100b12b0 legacy=0x7f5b101d7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.050 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:00:33.050 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 1 monclient: found mon.b 2024-03-29T11:00:33.050 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:33.050 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:33.050 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 1 -- 172.21.15.137:0/565383034 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b101d7ed0 con 0x7f5b1009ac40 2024-03-29T11:00:33.050 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient: _finish_auth 0 2024-03-29T11:00:33.050 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:00:33.051 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:33.051 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5b15c1e640 5 monclient: authenticate success, global_id 4117 2024-03-29T11:00:33.051 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5b15c1e640 10 monclient: _renew_subs 2024-03-29T11:00:33.051 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5b15c1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:33.051 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5b15c1e640 1 -- 172.21.15.137:0/565383034 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b101e3400 con 0x7f5b1009ac40 2024-03-29T11:00:33.051 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5b15c1e640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:33.051 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5b15c1e640 10 monclient: _renew_subs 2024-03-29T11:00:33.051 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5b15c1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:33.051 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5b15c1e640 1 -- 172.21.15.137:0/565383034 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b101e3770 con 0x7f5b1009ac40 2024-03-29T11:00:33.051 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5b15c1e640 1 librados: init done 2024-03-29T11:00:33.051 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 1 -- 172.21.15.137:0/565383034 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b00004bf0 con 0x7f5b1009ac40 2024-03-29T11:00:33.051 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:33.051 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 1 -- 172.21.15.137:0/565383034 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f5b00004f00 con 0x7f5b1009ac40 2024-03-29T11:00:33.051 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.052 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:33.052 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 monclient: dump: 2024-03-29T11:00:33.052 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.052 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.052 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.052 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.052 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.052 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.052 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.052 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.052 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.052 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.052 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5b0dffb640 4 set_mon_vals no callback set 2024-03-29T11:00:33.052 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 1 -- 172.21.15.137:0/565383034 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5b00005470 con 0x7f5b1009ac40 2024-03-29T11:00:33.052 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 1 -- 172.21.15.137:0/565383034 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 480390900 0 0) 0x7f5b000068a0 con 0x7f5b1009ac40 2024-03-29T11:00:33.052 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 10 client.4117.objecter ms_dispatch 0x7f5b10000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:00:33.053 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 3 client.4117.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:00:33.053 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 3 client.4117.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:00:33.053 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.051+0000 7f5af7fff640 20 client.4117.objecter dump_active .. 0 homeless 2024-03-29T11:00:33.053 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.052+0000 7f5b15c1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:33.053 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.052+0000 7f5b15c1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:33.053 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.052+0000 7f5b15c1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:33.053 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.052+0000 7f5b15c1e640 1 -- 172.21.15.137:0/565383034 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b10143100 con 0x7f5b1009ac40 2024-03-29T11:00:33.053 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.054+0000 7f5af7fff640 1 -- 172.21.15.137:0/565383034 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5b00006a60 con 0x7f5b1009ac40 2024-03-29T11:00:33.053 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.054+0000 7f5af7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:33.053 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.054+0000 7f5af7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:33.107 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.127+0000 7f5b15c1e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5c6efb84-6592-41d7-b4f1-b4a33d2d2f83", "id": 1}] 2024-03-29T11:00:33.107 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.127+0000 7f5b15c1e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5c6efb84-6592-41d7-b4f1-b4a33d2d2f83", "id": 1}] 2024-03-29T11:00:33.107 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.127+0000 7f5b15c1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:33.107 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.127+0000 7f5b15c1e640 1 -- 172.21.15.137:0/565383034 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd new", "uuid": "5c6efb84-6592-41d7-b4f1-b4a33d2d2f83", "id": 1} v 0) v1 -- 0x7f5b1013dfe0 con 0x7f5b1009ac40 2024-03-29T11:00:33.111 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.131+0000 7f5af7fff640 1 -- 172.21.15.137:0/565383034 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5c6efb84-6592-41d7-b4f1-b4a33d2d2f83", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2437024338 0 940483276) 0x7f5b00005890 con 0x7f5b1009ac40 2024-03-29T11:00:33.111 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.131+0000 7f5af7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5c6efb84-6592-41d7-b4f1-b4a33d2d2f83", "id": 1}] 2024-03-29T11:00:33.111 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.131+0000 7f5af7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:33.111 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-03-29T11:00:33.112 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.131+0000 7f5b15c1e640 10 librados: watch_flush enter 2024-03-29T11:00:33.113 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.131+0000 7f5b15c1e640 10 librados: watch_flush exit 2024-03-29T11:00:33.113 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.131+0000 7f5b15c1e640 20 client.4117.objecter shutdown clearing up homeless session... 2024-03-29T11:00:33.113 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.131+0000 7f5b15c1e640 10 client.4117.objecter successfully canceled tick 2024-03-29T11:00:33.113 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.132+0000 7f5b15c1e640 10 monclient: shutdown 2024-03-29T11:00:33.113 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.132+0000 7f5b15c1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:33.113 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.132+0000 7f5b15c1e640 1 -- 172.21.15.137:0/565383034 >> v1:172.21.15.137:6789/0 conn(0x7f5b1009ac40 legacy=0x7f5b10142860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.113 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.132+0000 7f5b15c1e640 1 -- 172.21.15.137:0/565383034 shutdown_connections 2024-03-29T11:00:33.113 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.132+0000 7f5b15c1e640 1 -- 172.21.15.137:0/565383034 >> 172.21.15.137:0/565383034 conn(0x7f5b10066b00 msgr2=0x7f5b1013d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:33.113 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.132+0000 7f5b15c1e640 1 -- 172.21.15.137:0/565383034 shutdown_connections 2024-03-29T11:00:33.113 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.132+0000 7f5b15c1e640 1 -- 172.21.15.137:0/565383034 wait complete. 2024-03-29T11:00:33.113 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.132+0000 7f5b15c1e640 1 librados: shutdown 2024-03-29T11:00:33.122 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05 2 2024-03-29T11:00:33.223 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.242+0000 7f29f279d640 10 monclient: get_monmap_and_config 2024-03-29T11:00:33.223 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.242+0000 7f29f279d640 10 monclient: build_initial_monmap 2024-03-29T11:00:33.223 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.242+0000 7f29f279d640 10 monclient: monmap: 2024-03-29T11:00:33.223 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 0 2024-03-29T11:00:33.223 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.223 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T11:00:33.243963+0000 2024-03-29T11:00:33.224 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T11:00:33.243963+0000 2024-03-29T11:00:33.224 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:33.224 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.224 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:33.224 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:00:33.224 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:00:33.224 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.224 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.243+0000 7f29f279d640 1 Processor -- start 2024-03-29T11:00:33.224 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.243+0000 7f29f279d640 1 -- start start 2024-03-29T11:00:33.224 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.243+0000 7f29f279d640 10 monclient: init 2024-03-29T11:00:33.233 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29f279d640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:33.233 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29f279d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:00:33.233 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29f279d640 10 monclient(hunting): picked mon.noname-c con 0x7f29ec09ac40 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:33.233 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29f279d640 10 monclient(hunting): picked mon.noname-a con 0x7f29ec09c870 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:33.233 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29f279d640 10 monclient(hunting): picked mon.noname-b con 0x7f29ec0b4360 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:33.233 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29f279d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29ec09dec0 con 0x7f29ec09c870 2024-03-29T11:00:33.233 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29f279d640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29ec09da30 con 0x7f29ec09ac40 2024-03-29T11:00:33.233 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29f279d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29ec0b39c0 con 0x7f29ec0b4360 2024-03-29T11:00:33.234 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29f279d640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:33.234 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29f279d640 10 monclient(hunting): authenticate will time out at 778.578192s 2024-03-29T11:00:33.234 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29ebfff640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f29ec09ac40 0x7f29ec0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.137:46870/0 (socket says 172.21.15.137:46870) 2024-03-29T11:00:33.234 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29ebfff640 1 -- 172.21.15.137:0/1163300885 learned_addr learned my addr 172.21.15.137:0/1163300885 (peer_addr_for_me v1:172.21.15.137:0/0) 2024-03-29T11:00:33.234 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 1 -- 172.21.15.137:0/1163300885 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658200698 0 0) 0x7f29ec09da30 con 0x7f29ec09ac40 2024-03-29T11:00:33.234 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.234 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.234 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.234 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:00:33.234 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 1 -- 172.21.15.137:0/1163300885 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29d4003520 con 0x7f29ec09ac40 2024-03-29T11:00:33.234 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 1 -- 172.21.15.137:0/1163300885 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3511502521 0 0) 0x7f29dc001110 con 0x7f29ec09c870 2024-03-29T11:00:33.234 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.234 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.234 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.235 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:00:33.235 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 1 -- 172.21.15.137:0/1163300885 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29ec09da30 con 0x7f29ec09c870 2024-03-29T11:00:33.235 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 1 -- 172.21.15.137:0/1163300885 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1271794731 0 0) 0x7f29ec0b39c0 con 0x7f29ec0b4360 2024-03-29T11:00:33.235 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.235 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.235 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.235 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:00:33.235 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 1 -- 172.21.15.137:0/1163300885 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29dc001110 con 0x7f29ec0b4360 2024-03-29T11:00:33.235 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 1 -- 172.21.15.137:0/1163300885 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 544353561 0 0) 0x7f29d4003520 con 0x7f29ec09ac40 2024-03-29T11:00:33.235 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 1 -- 172.21.15.137:0/1163300885 >> v1:172.21.15.71:6790/0 conn(0x7f29ec0b4360 legacy=0x7f29ec14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.235 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 1 -- 172.21.15.137:0/1163300885 >> v1:172.21.15.71:6789/0 conn(0x7f29ec09c870 legacy=0x7f29ec13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.235 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:00:33.235 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 1 monclient: found mon.noname-c 2024-03-29T11:00:33.235 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:33.235 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.137:6789/0 2024-03-29T11:00:33.236 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 1 -- 172.21.15.137:0/1163300885 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ec0b5160 con 0x7f29ec09ac40 2024-03-29T11:00:33.236 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 10 monclient: _finish_auth 0 2024-03-29T11:00:33.236 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:00:33.236 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:33.236 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29f279d640 5 monclient: authenticate success, global_id 4120 2024-03-29T11:00:33.236 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 1 -- 172.21.15.137:0/1163300885 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f29e0001690 con 0x7f29ec09ac40 2024-03-29T11:00:33.236 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.236 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:33.236 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 10 monclient: dump: 2024-03-29T11:00:33.236 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.236 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.236 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.236 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.236 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.236 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.237 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.237 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.237 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.237 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.237 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29f279d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:33.237 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 1 -- 172.21.15.137:0/1163300885 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29e0001870 con 0x7f29ec09ac40 2024-03-29T11:00:33.237 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:33.237 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29f279d640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:33.237 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 1 -- 172.21.15.137:0/1163300885 <== mon.1 v1:172.21.15.137:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f29e0001c80 con 0x7f29ec09ac40 2024-03-29T11:00:33.237 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29f279d640 4 set_mon_vals no callback set 2024-03-29T11:00:33.237 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.237 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:33.237 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.244+0000 7f29eaffd640 10 monclient: dump: 2024-03-29T11:00:33.237 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.238 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.238 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.238 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.238 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.238 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.238 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.238 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.238 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.238 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.238 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.245+0000 7f29f279d640 10 monclient: shutdown 2024-03-29T11:00:33.238 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.245+0000 7f29f279d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:33.238 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.245+0000 7f29f279d640 1 -- 172.21.15.137:0/1163300885 >> v1:172.21.15.137:6789/0 conn(0x7f29ec09ac40 legacy=0x7f29ec0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.238 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.245+0000 7f29f279d640 1 -- 172.21.15.137:0/1163300885 shutdown_connections 2024-03-29T11:00:33.238 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.245+0000 7f29f279d640 1 -- 172.21.15.137:0/1163300885 >> 172.21.15.137:0/1163300885 conn(0x7f29ec066b00 msgr2=0x7f29ec066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:33.238 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.245+0000 7f29f279d640 1 -- 172.21.15.137:0/1163300885 shutdown_connections 2024-03-29T11:00:33.239 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.245+0000 7f29f279d640 1 -- 172.21.15.137:0/1163300885 wait complete. 2024-03-29T11:00:33.239 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.246+0000 7f29f279d640 10 monclient: build_initial_monmap 2024-03-29T11:00:33.239 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.246+0000 7f29f279d640 10 monclient: monmap: 2024-03-29T11:00:33.239 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 0 2024-03-29T11:00:33.239 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:33.239 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 0.000000 2024-03-29T11:00:33.239 INFO:teuthology.orchestra.run.smithi137.stderr:created 0.000000 2024-03-29T11:00:33.239 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:33.239 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.239 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:33.239 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:00:33.239 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:00:33.239 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.239 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.246+0000 7f29f279d640 1 librados: starting msgr at 2024-03-29T11:00:33.240 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.246+0000 7f29f279d640 1 librados: starting objecter 2024-03-29T11:00:33.240 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.246+0000 7f29f279d640 1 Processor -- start 2024-03-29T11:00:33.240 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.246+0000 7f29f279d640 1 -- start start 2024-03-29T11:00:33.240 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.246+0000 7f29f279d640 1 librados: setting wanted keys 2024-03-29T11:00:33.240 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.246+0000 7f29f279d640 1 librados: calling monclient init 2024-03-29T11:00:33.240 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.246+0000 7f29f279d640 10 monclient: init 2024-03-29T11:00:33.240 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29f279d640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:33.240 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29f279d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:00:33.240 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29f279d640 10 monclient(hunting): picked mon.noname-b con 0x7f29ec09ac40 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:33.240 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29f279d640 10 monclient(hunting): picked mon.noname-a con 0x7f29ec0b1280 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:33.240 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29f279d640 10 monclient(hunting): picked mon.noname-c con 0x7f29ec1d6ed0 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:33.240 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29f279d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29ec1422a0 con 0x7f29ec0b1280 2024-03-29T11:00:33.240 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29f279d640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29ec142f50 con 0x7f29ec09ac40 2024-03-29T11:00:33.240 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29f279d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29ec1ccab0 con 0x7f29ec1d6ed0 2024-03-29T11:00:33.240 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29f279d640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:33.241 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29f279d640 10 monclient(hunting): authenticate will time out at 778.581310s 2024-03-29T11:00:33.241 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29ebfff640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f29ec09ac40 0x7f29ec141ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.137:46878/0 (socket says 172.21.15.137:46878) 2024-03-29T11:00:33.241 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29ebfff640 1 -- 172.21.15.137:0/3096065371 learned_addr learned my addr 172.21.15.137:0/3096065371 (peer_addr_for_me v1:172.21.15.137:0/0) 2024-03-29T11:00:33.241 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 10 client.?.objecter ms_handle_connect 0x7f29ec09ac40 2024-03-29T11:00:33.241 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:33.241 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 1 -- 172.21.15.137:0/3096065371 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1851833097 0 0) 0x7f29ec142f50 con 0x7f29ec09ac40 2024-03-29T11:00:33.241 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.241 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.241 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.241 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:00:33.241 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 1 -- 172.21.15.137:0/3096065371 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29d8004eb0 con 0x7f29ec09ac40 2024-03-29T11:00:33.241 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 10 client.?.objecter ms_handle_connect 0x7f29ec1d6ed0 2024-03-29T11:00:33.241 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:33.241 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 10 client.?.objecter ms_handle_connect 0x7f29ec0b1280 2024-03-29T11:00:33.241 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:33.242 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 1 -- 172.21.15.137:0/3096065371 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2606181168 0 0) 0x7f29ec1ccab0 con 0x7f29ec1d6ed0 2024-03-29T11:00:33.242 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.242 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.242 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.242 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:00:33.242 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 1 -- 172.21.15.137:0/3096065371 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29ec142f50 con 0x7f29ec1d6ed0 2024-03-29T11:00:33.242 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 1 -- 172.21.15.137:0/3096065371 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2417702079 0 0) 0x7f29ec1422a0 con 0x7f29ec0b1280 2024-03-29T11:00:33.242 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.242 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.242 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.242 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:00:33.242 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 1 -- 172.21.15.137:0/3096065371 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29ec1ccab0 con 0x7f29ec0b1280 2024-03-29T11:00:33.242 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 1 -- 172.21.15.137:0/3096065371 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3967492410 0 0) 0x7f29d8004eb0 con 0x7f29ec09ac40 2024-03-29T11:00:33.242 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 1 -- 172.21.15.137:0/3096065371 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29ec1422a0 con 0x7f29ec09ac40 2024-03-29T11:00:33.242 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 1 -- 172.21.15.137:0/3096065371 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f29e0001c40 con 0x7f29ec09ac40 2024-03-29T11:00:33.243 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.243 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:33.243 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 10 monclient(hunting): dump: 2024-03-29T11:00:33.243 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.243 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.243 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.243 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.243 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.243 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.243 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.243 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.243 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.243 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.243 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:33.243 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 1 -- 172.21.15.137:0/3096065371 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1828659542 0 0) 0x7f29ec142f50 con 0x7f29ec1d6ed0 2024-03-29T11:00:33.244 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.247+0000 7f29e8ff9640 1 -- 172.21.15.137:0/3096065371 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29d8004eb0 con 0x7f29ec1d6ed0 2024-03-29T11:00:33.244 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 1 -- 172.21.15.137:0/3096065371 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f29e4001b50 con 0x7f29ec1d6ed0 2024-03-29T11:00:33.244 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.244 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:33.244 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 10 monclient(hunting): dump: 2024-03-29T11:00:33.244 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.244 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.244 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.244 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.244 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.244 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.244 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.244 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.244 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.244 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.245 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 1 -- 172.21.15.137:0/3096065371 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2197675003 0 0) 0x7f29ec1ccab0 con 0x7f29ec0b1280 2024-03-29T11:00:33.245 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 1 -- 172.21.15.137:0/3096065371 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29ec142f50 con 0x7f29ec0b1280 2024-03-29T11:00:33.245 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 1 -- 172.21.15.137:0/3096065371 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 645819726 0 0) 0x7f29ec1422a0 con 0x7f29ec09ac40 2024-03-29T11:00:33.245 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 1 -- 172.21.15.137:0/3096065371 >> v1:172.21.15.71:6790/0 conn(0x7f29ec1d6ed0 legacy=0x7f29ec13adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.245 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 1 -- 172.21.15.137:0/3096065371 >> v1:172.21.15.71:6789/0 conn(0x7f29ec0b1280 legacy=0x7f29ec142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.245 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:00:33.245 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 1 monclient: found mon.b 2024-03-29T11:00:33.245 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:33.245 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:33.245 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 1 -- 172.21.15.137:0/3096065371 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ec1ccc40 con 0x7f29ec09ac40 2024-03-29T11:00:33.245 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:00:33.245 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:00:33.245 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:33.245 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29f279d640 5 monclient: authenticate success, global_id 4123 2024-03-29T11:00:33.245 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29f279d640 10 monclient: _renew_subs 2024-03-29T11:00:33.246 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29f279d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:33.246 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29f279d640 1 -- 172.21.15.137:0/3096065371 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29ec13b4d0 con 0x7f29ec09ac40 2024-03-29T11:00:33.246 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29f279d640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:33.246 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29f279d640 10 monclient: _renew_subs 2024-03-29T11:00:33.246 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29f279d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:33.246 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 1 -- 172.21.15.137:0/3096065371 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29e0004bf0 con 0x7f29ec09ac40 2024-03-29T11:00:33.246 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29f279d640 1 -- 172.21.15.137:0/3096065371 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29ec1e3720 con 0x7f29ec09ac40 2024-03-29T11:00:33.246 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29f279d640 1 librados: init done 2024-03-29T11:00:33.246 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:33.246 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 1 -- 172.21.15.137:0/3096065371 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f29e0004f00 con 0x7f29ec09ac40 2024-03-29T11:00:33.246 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.246 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:33.246 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 10 monclient: dump: 2024-03-29T11:00:33.246 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.247 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.247 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.247 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.247 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.247 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.247 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.247 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.247 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.247 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.247 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29ea7fc640 4 set_mon_vals no callback set 2024-03-29T11:00:33.247 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 1 -- 172.21.15.137:0/3096065371 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f29e00053f0 con 0x7f29ec09ac40 2024-03-29T11:00:33.247 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 1 -- 172.21.15.137:0/3096065371 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 142901000 0 0) 0x7f29e0005ac0 con 0x7f29ec09ac40 2024-03-29T11:00:33.247 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 10 client.4123.objecter ms_dispatch 0x7f29ec000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:00:33.247 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 3 client.4123.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:00:33.247 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 3 client.4123.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:00:33.248 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.248+0000 7f29e8ff9640 20 client.4123.objecter dump_active .. 0 homeless 2024-03-29T11:00:33.248 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.249+0000 7f29f279d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:33.248 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.249+0000 7f29f279d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:33.248 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.249+0000 7f29f279d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:33.248 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.249+0000 7f29f279d640 1 -- 172.21.15.137:0/3096065371 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29cc005080 con 0x7f29ec09ac40 2024-03-29T11:00:33.248 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.251+0000 7f29e8ff9640 1 -- 172.21.15.137:0/3096065371 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f29e0005da0 con 0x7f29ec09ac40 2024-03-29T11:00:33.248 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.251+0000 7f29e8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:33.248 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.251+0000 7f29e8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:33.248 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T11:00:33.253+0000 7fab49e36040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:00:33.250 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:00:33.269+0000 7f5976230040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:00:33.290 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T11:00:33.310+0000 7fab49e36040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:00:33.303 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:00:33.322+0000 7f5976230040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:00:33.303 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.323+0000 7f29f279d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05", "id": 2}] 2024-03-29T11:00:33.303 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.323+0000 7f29f279d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05", "id": 2}] 2024-03-29T11:00:33.304 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.323+0000 7f29f279d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:33.304 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.323+0000 7f29f279d640 1 -- 172.21.15.137:0/3096065371 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05", "id": 2} v 0) v1 -- 0x7f29cc0058b0 con 0x7f29ec09ac40 2024-03-29T11:00:33.307 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.327+0000 7f29e8ff9640 1 -- 172.21.15.137:0/3096065371 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 4222751079 0 216599125) 0x7f29e0005770 con 0x7f29ec09ac40 2024-03-29T11:00:33.307 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.327+0000 7f29e8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05", "id": 2}] 2024-03-29T11:00:33.307 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.327+0000 7f29e8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:33.307 INFO:teuthology.orchestra.run.smithi137.stdout:2 2024-03-29T11:00:33.309 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.328+0000 7f29f279d640 10 librados: watch_flush enter 2024-03-29T11:00:33.309 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.328+0000 7f29f279d640 10 librados: watch_flush exit 2024-03-29T11:00:33.309 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.328+0000 7f29f279d640 20 client.4123.objecter shutdown clearing up homeless session... 2024-03-29T11:00:33.309 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.328+0000 7f29f279d640 10 client.4123.objecter successfully canceled tick 2024-03-29T11:00:33.309 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.328+0000 7f29f279d640 10 monclient: shutdown 2024-03-29T11:00:33.309 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.328+0000 7f29f279d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:33.309 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.328+0000 7f29f279d640 1 -- 172.21.15.137:0/3096065371 >> v1:172.21.15.137:6789/0 conn(0x7f29ec09ac40 legacy=0x7f29ec141ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.309 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.328+0000 7f29f279d640 1 -- 172.21.15.137:0/3096065371 shutdown_connections 2024-03-29T11:00:33.309 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.328+0000 7f29f279d640 1 -- 172.21.15.137:0/3096065371 >> 172.21.15.137:0/3096065371 conn(0x7f29ec066b00 msgr2=0x7f29ec066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:33.309 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.328+0000 7f29f279d640 1 -- 172.21.15.137:0/3096065371 shutdown_connections 2024-03-29T11:00:33.309 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.328+0000 7f29f279d640 1 -- 172.21.15.137:0/3096065371 wait complete. 2024-03-29T11:00:33.309 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.328+0000 7f29f279d640 1 librados: shutdown 2024-03-29T11:00:33.319 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new 395ce08b-6e75-4493-ad8b-3ca9763c444e 3 2024-03-29T11:00:33.337 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T11:00:33.357+0000 7fab49e36040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:00:33.350 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:00:33.368+0000 7f5976230040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:00:33.394 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T11:00:33.414+0000 7fab49e36040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:00:33.401 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:00:33.421+0000 7f5976230040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:00:33.418 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.438+0000 7f2a1f782640 10 monclient: get_monmap_and_config 2024-03-29T11:00:33.419 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.438+0000 7f2a1f782640 10 monclient: build_initial_monmap 2024-03-29T11:00:33.419 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.438+0000 7f2a1f782640 10 monclient: monmap: 2024-03-29T11:00:33.419 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 0 2024-03-29T11:00:33.419 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.419 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T11:00:33.439952+0000 2024-03-29T11:00:33.419 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T11:00:33.439952+0000 2024-03-29T11:00:33.419 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:33.419 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.419 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:33.419 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:00:33.419 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:00:33.419 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.433 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.439+0000 7f2a1f782640 1 Processor -- start 2024-03-29T11:00:33.434 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.439+0000 7f2a1f782640 1 -- start start 2024-03-29T11:00:33.434 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.439+0000 7f2a1f782640 10 monclient: init 2024-03-29T11:00:33.434 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a1f782640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:33.434 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a1f782640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:00:33.434 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a1f782640 10 monclient(hunting): picked mon.noname-b con 0x7f2a1809ac40 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:33.434 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a1f782640 10 monclient(hunting): picked mon.noname-a con 0x7f2a1809c870 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:33.434 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a1f782640 10 monclient(hunting): picked mon.noname-c con 0x7f2a180b4360 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:33.434 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a1f782640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a1809dec0 con 0x7f2a1809c870 2024-03-29T11:00:33.434 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a1f782640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a1809da30 con 0x7f2a180b4360 2024-03-29T11:00:33.434 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a1f782640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a180b39c0 con 0x7f2a1809ac40 2024-03-29T11:00:33.434 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a1f782640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:33.434 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a1f782640 10 monclient(hunting): authenticate will time out at 778.774309s 2024-03-29T11:00:33.434 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a1dd1f640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f2a180b4360 0x7f2a1814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.137:46884/0 (socket says 172.21.15.137:46884) 2024-03-29T11:00:33.435 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a1dd1f640 1 -- 172.21.15.137:0/545464520 learned_addr learned my addr 172.21.15.137:0/545464520 (peer_addr_for_me v1:172.21.15.137:0/0) 2024-03-29T11:00:33.435 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 1 -- 172.21.15.137:0/545464520 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1294616741 0 0) 0x7f2a1809da30 con 0x7f2a180b4360 2024-03-29T11:00:33.435 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.435 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.435 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.435 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:00:33.435 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 1 -- 172.21.15.137:0/545464520 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a04003520 con 0x7f2a180b4360 2024-03-29T11:00:33.435 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 1 -- 172.21.15.137:0/545464520 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1259509325 0 0) 0x7f2a14001110 con 0x7f2a1809c870 2024-03-29T11:00:33.435 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.435 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.435 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.435 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:00:33.436 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 1 -- 172.21.15.137:0/545464520 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a1809da30 con 0x7f2a1809c870 2024-03-29T11:00:33.436 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 1 -- 172.21.15.137:0/545464520 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1724083409 0 0) 0x7f2a180b39c0 con 0x7f2a1809ac40 2024-03-29T11:00:33.436 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.436 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.436 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.436 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:00:33.436 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 1 -- 172.21.15.137:0/545464520 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a14001110 con 0x7f2a1809ac40 2024-03-29T11:00:33.436 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 1 -- 172.21.15.137:0/545464520 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1362646979 0 0) 0x7f2a04003520 con 0x7f2a180b4360 2024-03-29T11:00:33.436 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 1 -- 172.21.15.137:0/545464520 >> v1:172.21.15.71:6790/0 conn(0x7f2a1809ac40 legacy=0x7f2a180b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.436 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 1 -- 172.21.15.137:0/545464520 >> v1:172.21.15.71:6789/0 conn(0x7f2a1809c870 legacy=0x7f2a1813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.436 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:00:33.436 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 1 monclient: found mon.noname-c 2024-03-29T11:00:33.436 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:33.436 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.137:6789/0 2024-03-29T11:00:33.437 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 1 -- 172.21.15.137:0/545464520 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a180b5160 con 0x7f2a180b4360 2024-03-29T11:00:33.437 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 10 monclient: _finish_auth 0 2024-03-29T11:00:33.437 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:00:33.437 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:33.437 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a1f782640 5 monclient: authenticate success, global_id 4126 2024-03-29T11:00:33.437 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 1 -- 172.21.15.137:0/545464520 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2a10001a10 con 0x7f2a180b4360 2024-03-29T11:00:33.437 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a1f782640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:33.437 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.437 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:33.437 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a0ffff640 10 monclient: dump: 2024-03-29T11:00:33.437 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.437 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.437 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.438 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.438 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.438 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.438 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.438 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.439 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.439 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.439 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.440+0000 7f2a1f782640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:33.439 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.441+0000 7f2a0ffff640 1 -- 172.21.15.137:0/545464520 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a10001b70 con 0x7f2a180b4360 2024-03-29T11:00:33.439 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.441+0000 7f2a0ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:33.439 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.441+0000 7f2a1f782640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:33.439 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.441+0000 7f2a0ffff640 1 -- 172.21.15.137:0/545464520 <== mon.1 v1:172.21.15.137:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2a10001f80 con 0x7f2a180b4360 2024-03-29T11:00:33.439 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.441+0000 7f2a0ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.439 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.441+0000 7f2a1f782640 4 set_mon_vals no callback set 2024-03-29T11:00:33.440 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.441+0000 7f2a0ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:33.440 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.441+0000 7f2a0ffff640 10 monclient: dump: 2024-03-29T11:00:33.440 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.440 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.440 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.440 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.440 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.440 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.440 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.440 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.440 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.440 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.440 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.441+0000 7f2a1f782640 10 monclient: shutdown 2024-03-29T11:00:33.441 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.441+0000 7f2a1f782640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:33.441 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.441+0000 7f2a1f782640 1 -- 172.21.15.137:0/545464520 >> v1:172.21.15.137:6789/0 conn(0x7f2a180b4360 legacy=0x7f2a1814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.441 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.441+0000 7f2a1f782640 1 -- 172.21.15.137:0/545464520 shutdown_connections 2024-03-29T11:00:33.441 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.441+0000 7f2a1f782640 1 -- 172.21.15.137:0/545464520 >> 172.21.15.137:0/545464520 conn(0x7f2a18066b00 msgr2=0x7f2a18066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:33.441 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.441+0000 7f2a1f782640 1 -- 172.21.15.137:0/545464520 shutdown_connections 2024-03-29T11:00:33.441 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.441+0000 7f2a1f782640 1 -- 172.21.15.137:0/545464520 wait complete. 2024-03-29T11:00:33.441 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.442+0000 7f2a1f782640 10 monclient: build_initial_monmap 2024-03-29T11:00:33.441 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.442+0000 7f2a1f782640 10 monclient: monmap: 2024-03-29T11:00:33.441 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 0 2024-03-29T11:00:33.441 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:33.441 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 0.000000 2024-03-29T11:00:33.441 INFO:teuthology.orchestra.run.smithi137.stderr:created 0.000000 2024-03-29T11:00:33.441 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:33.442 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.442 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:33.442 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:00:33.442 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:00:33.442 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.442 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.442+0000 7f2a1f782640 1 librados: starting msgr at 2024-03-29T11:00:33.442 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.442+0000 7f2a1f782640 1 librados: starting objecter 2024-03-29T11:00:33.442 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.442+0000 7f2a1f782640 1 Processor -- start 2024-03-29T11:00:33.442 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.442+0000 7f2a1f782640 1 -- start start 2024-03-29T11:00:33.442 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.442+0000 7f2a1f782640 1 librados: setting wanted keys 2024-03-29T11:00:33.442 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.442+0000 7f2a1f782640 1 librados: calling monclient init 2024-03-29T11:00:33.442 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.442+0000 7f2a1f782640 10 monclient: init 2024-03-29T11:00:33.443 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a1f782640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:33.443 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a1f782640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:00:33.443 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a1f782640 10 monclient(hunting): picked mon.noname-a con 0x7f2a1809ac40 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:33.443 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a1f782640 10 monclient(hunting): picked mon.noname-c con 0x7f2a180afba0 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:33.443 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a1f782640 10 monclient(hunting): picked mon.noname-b con 0x7f2a18140b40 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:33.443 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a1f782640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a18140030 con 0x7f2a1809ac40 2024-03-29T11:00:33.443 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a1f782640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a181d7280 con 0x7f2a18140b40 2024-03-29T11:00:33.443 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a1f782640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a18141d10 con 0x7f2a180afba0 2024-03-29T11:00:33.443 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a1f782640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:33.443 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a1f782640 10 monclient(hunting): authenticate will time out at 778.777153s 2024-03-29T11:00:33.443 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a1dd1f640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f2a18140b40 0x7f2a18138da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.137:46886/0 (socket says 172.21.15.137:46886) 2024-03-29T11:00:33.443 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a1dd1f640 1 -- 172.21.15.137:0/1245779429 learned_addr learned my addr 172.21.15.137:0/1245779429 (peer_addr_for_me v1:172.21.15.137:0/0) 2024-03-29T11:00:33.443 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 client.?.objecter ms_handle_connect 0x7f2a18140b40 2024-03-29T11:00:33.443 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:33.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 client.?.objecter ms_handle_connect 0x7f2a180afba0 2024-03-29T11:00:33.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:33.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 client.?.objecter ms_handle_connect 0x7f2a1809ac40 2024-03-29T11:00:33.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:33.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 1 -- 172.21.15.137:0/1245779429 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093141153 0 0) 0x7f2a181d7280 con 0x7f2a18140b40 2024-03-29T11:00:33.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:00:33.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 1 -- 172.21.15.137:0/1245779429 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29fc004eb0 con 0x7f2a18140b40 2024-03-29T11:00:33.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 1 -- 172.21.15.137:0/1245779429 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936236862 0 0) 0x7f2a18141d10 con 0x7f2a180afba0 2024-03-29T11:00:33.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.445 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:00:33.445 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 1 -- 172.21.15.137:0/1245779429 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a181d7280 con 0x7f2a180afba0 2024-03-29T11:00:33.445 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 1 -- 172.21.15.137:0/1245779429 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2247410114 0 0) 0x7f2a18140030 con 0x7f2a1809ac40 2024-03-29T11:00:33.445 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.445 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.445 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.445 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:00:33.445 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 1 -- 172.21.15.137:0/1245779429 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a18141d10 con 0x7f2a1809ac40 2024-03-29T11:00:33.445 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 1 -- 172.21.15.137:0/1245779429 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1668583317 0 0) 0x7f29fc004eb0 con 0x7f2a18140b40 2024-03-29T11:00:33.445 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 1 -- 172.21.15.137:0/1245779429 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a18140030 con 0x7f2a18140b40 2024-03-29T11:00:33.445 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 1 -- 172.21.15.137:0/1245779429 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2a10001ec0 con 0x7f2a18140b40 2024-03-29T11:00:33.445 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:33.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 monclient(hunting): dump: 2024-03-29T11:00:33.446 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.446 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.446 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.446 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.446 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.446 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.446 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.446 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.446 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.446 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:33.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 1 -- 172.21.15.137:0/1245779429 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 221788265 0 0) 0x7f2a18141d10 con 0x7f2a1809ac40 2024-03-29T11:00:33.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 1 -- 172.21.15.137:0/1245779429 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29fc004eb0 con 0x7f2a1809ac40 2024-03-29T11:00:33.447 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 1 -- 172.21.15.137:0/1245779429 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1753606246 0 0) 0x7f2a181d7280 con 0x7f2a180afba0 2024-03-29T11:00:33.447 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 1 -- 172.21.15.137:0/1245779429 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a18141d10 con 0x7f2a180afba0 2024-03-29T11:00:33.447 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 1 -- 172.21.15.137:0/1245779429 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2a08001870 con 0x7f2a1809ac40 2024-03-29T11:00:33.447 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.447 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:33.447 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 monclient(hunting): dump: 2024-03-29T11:00:33.447 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.447 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.447 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.447 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.447 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.447 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.447 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.448 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.448 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.448 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.448 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 1 -- 172.21.15.137:0/1245779429 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2a140018e0 con 0x7f2a180afba0 2024-03-29T11:00:33.448 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.448 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:33.448 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 monclient(hunting): dump: 2024-03-29T11:00:33.448 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.448 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.448 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.448 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.448 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.448 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.448 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.448 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.449 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.449 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.449 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 1 -- 172.21.15.137:0/1245779429 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3011629836 0 0) 0x7f2a18140030 con 0x7f2a18140b40 2024-03-29T11:00:33.449 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 1 -- 172.21.15.137:0/1245779429 >> v1:172.21.15.71:6790/0 conn(0x7f2a180afba0 legacy=0x7f2a181d77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.449 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 1 -- 172.21.15.137:0/1245779429 >> v1:172.21.15.71:6789/0 conn(0x7f2a1809ac40 legacy=0x7f2a18141600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.449 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:00:33.449 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 1 monclient: found mon.b 2024-03-29T11:00:33.449 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:33.449 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:33.449 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 1 -- 172.21.15.137:0/1245779429 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a181d7eb0 con 0x7f2a18140b40 2024-03-29T11:00:33.449 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 monclient: _finish_auth 0 2024-03-29T11:00:33.449 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:00:33.449 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:33.449 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.443+0000 7f2a1f782640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:00:33.450 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.444+0000 7f2a1f782640 10 monclient: _renew_subs 2024-03-29T11:00:33.450 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.444+0000 7f2a1f782640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:33.450 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.444+0000 7f2a1f782640 1 -- 172.21.15.137:0/1245779429 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a181394b0 con 0x7f2a18140b40 2024-03-29T11:00:33.450 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.444+0000 7f2a0dffb640 1 -- 172.21.15.137:0/1245779429 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a10004e10 con 0x7f2a18140b40 2024-03-29T11:00:33.450 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.444+0000 7f2a0dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:33.450 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.444+0000 7f2a0dffb640 1 -- 172.21.15.137:0/1245779429 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2a10005120 con 0x7f2a18140b40 2024-03-29T11:00:33.450 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.444+0000 7f2a0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.450 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.444+0000 7f2a0dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:33.450 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.444+0000 7f2a0dffb640 10 monclient: dump: 2024-03-29T11:00:33.450 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.450 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.450 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.450 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.450 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.451 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.451 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.451 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.451 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.451 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.451 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.444+0000 7f2a0f7fe640 4 set_mon_vals no callback set 2024-03-29T11:00:33.451 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.444+0000 7f2a1f782640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:33.451 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.444+0000 7f2a0dffb640 1 -- 172.21.15.137:0/1245779429 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2a10005650 con 0x7f2a18140b40 2024-03-29T11:00:33.451 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.444+0000 7f2a1f782640 10 monclient: _renew_subs 2024-03-29T11:00:33.451 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.444+0000 7f2a1f782640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:33.451 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.444+0000 7f2a1f782640 1 -- 172.21.15.137:0/1245779429 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a181e3a10 con 0x7f2a18140b40 2024-03-29T11:00:33.451 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.444+0000 7f2a1f782640 1 librados: init done 2024-03-29T11:00:33.451 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.444+0000 7f2a0dffb640 1 -- 172.21.15.137:0/1245779429 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 1604169080 0 0) 0x7f2a10006010 con 0x7f2a18140b40 2024-03-29T11:00:33.451 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.444+0000 7f2a0dffb640 10 client.4129.objecter ms_dispatch 0x7f2a18000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:00:33.451 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.444+0000 7f2a0dffb640 3 client.4129.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:00:33.452 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.444+0000 7f2a0dffb640 3 client.4129.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:00:33.452 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.444+0000 7f2a0dffb640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-29T11:00:33.452 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.445+0000 7f2a1f782640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:33.452 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.445+0000 7f2a1f782640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:33.452 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.445+0000 7f2a1f782640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:33.452 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.445+0000 7f2a1f782640 1 -- 172.21.15.137:0/1245779429 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a181d70e0 con 0x7f2a18140b40 2024-03-29T11:00:33.452 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.447+0000 7f2a0dffb640 1 -- 172.21.15.137:0/1245779429 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2a181d70e0 con 0x7f2a18140b40 2024-03-29T11:00:33.452 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.447+0000 7f2a0dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:33.452 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.447+0000 7f2a0dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:33.453 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T11:00:33.459+0000 7fab49e36040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:00:33.453 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:00:33.464+0000 7f5976230040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:00:33.499 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.519+0000 7f2a1f782640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "395ce08b-6e75-4493-ad8b-3ca9763c444e", "id": 3}] 2024-03-29T11:00:33.500 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.519+0000 7f2a1f782640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "395ce08b-6e75-4493-ad8b-3ca9763c444e", "id": 3}] 2024-03-29T11:00:33.500 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.519+0000 7f2a1f782640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:33.500 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.519+0000 7f2a1f782640 1 -- 172.21.15.137:0/1245779429 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd new", "uuid": "395ce08b-6e75-4493-ad8b-3ca9763c444e", "id": 3} v 0) v1 -- 0x7f2a1809b020 con 0x7f2a18140b40 2024-03-29T11:00:33.503 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.523+0000 7f2a0dffb640 1 -- 172.21.15.137:0/1245779429 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "395ce08b-6e75-4493-ad8b-3ca9763c444e", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 2365837052 0 525046306) 0x7f2a10006320 con 0x7f2a18140b40 2024-03-29T11:00:33.503 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.523+0000 7f2a0dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "395ce08b-6e75-4493-ad8b-3ca9763c444e", "id": 3}] 2024-03-29T11:00:33.503 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.523+0000 7f2a0dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:33.504 INFO:teuthology.orchestra.run.smithi137.stdout:3 2024-03-29T11:00:33.506 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.524+0000 7f2a1f782640 10 librados: watch_flush enter 2024-03-29T11:00:33.506 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.524+0000 7f2a1f782640 10 librados: watch_flush exit 2024-03-29T11:00:33.506 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.524+0000 7f2a1f782640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-29T11:00:33.506 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.524+0000 7f2a1f782640 10 client.4129.objecter successfully canceled tick 2024-03-29T11:00:33.506 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.524+0000 7f2a1f782640 10 monclient: shutdown 2024-03-29T11:00:33.506 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.524+0000 7f2a1f782640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:33.506 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.524+0000 7f2a1f782640 1 -- 172.21.15.137:0/1245779429 >> v1:172.21.15.137:6789/0 conn(0x7f2a18140b40 legacy=0x7f2a18138da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.506 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.524+0000 7f2a1f782640 1 -- 172.21.15.137:0/1245779429 shutdown_connections 2024-03-29T11:00:33.507 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.524+0000 7f2a1f782640 1 -- 172.21.15.137:0/1245779429 >> 172.21.15.137:0/1245779429 conn(0x7f2a18066b00 msgr2=0x7f2a1813b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:33.507 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.524+0000 7f2a1f782640 1 -- 172.21.15.137:0/1245779429 shutdown_connections 2024-03-29T11:00:33.507 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.524+0000 7f2a1f782640 1 -- 172.21.15.137:0/1245779429 wait complete. 2024-03-29T11:00:33.507 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.524+0000 7f2a1f782640 1 librados: shutdown 2024-03-29T11:00:33.515 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new a9434f80-2148-42b6-a4d9-ca9b055b61c9 4 2024-03-29T11:00:33.559 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T11:00:33.579+0000 7fab49e36040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:00:33.561 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:00:33.580+0000 7f5976230040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:00:33.631 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:00:33.650+0000 7f5976230040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:00:33.633 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T11:00:33.653+0000 7fab49e36040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:00:33.655 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.675+0000 7fa2f978d640 10 monclient: get_monmap_and_config 2024-03-29T11:00:33.655 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.675+0000 7fa2f978d640 10 monclient: build_initial_monmap 2024-03-29T11:00:33.655 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.675+0000 7fa2f978d640 10 monclient: monmap: 2024-03-29T11:00:33.655 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 0 2024-03-29T11:00:33.655 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.656 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T11:00:33.676551+0000 2024-03-29T11:00:33.656 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T11:00:33.676551+0000 2024-03-29T11:00:33.656 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:33.656 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.656 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:33.656 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:00:33.656 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:00:33.656 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.666 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.676+0000 7fa2f978d640 1 Processor -- start 2024-03-29T11:00:33.666 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.676+0000 7fa2f978d640 1 -- start start 2024-03-29T11:00:33.666 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.676+0000 7fa2f978d640 10 monclient: init 2024-03-29T11:00:33.666 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.676+0000 7fa2f978d640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:33.666 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.676+0000 7fa2f978d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:00:33.666 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.676+0000 7fa2f978d640 10 monclient(hunting): picked mon.noname-b con 0x7fa2f40b8240 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:33.666 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.676+0000 7fa2f978d640 10 monclient(hunting): picked mon.noname-a con 0x7fa2f40b1f60 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:33.666 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.676+0000 7fa2f978d640 10 monclient(hunting): picked mon.noname-c con 0x7fa2f409c8c0 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:33.666 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.676+0000 7fa2f978d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f409bd80 con 0x7fa2f40b1f60 2024-03-29T11:00:33.667 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.676+0000 7fa2f978d640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f409b960 con 0x7fa2f409c8c0 2024-03-29T11:00:33.667 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.676+0000 7fa2f978d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f40b2ec0 con 0x7fa2f40b8240 2024-03-29T11:00:33.667 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.676+0000 7fa2f978d640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:33.667 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.676+0000 7fa2f978d640 10 monclient(hunting): authenticate will time out at 779.010755s 2024-03-29T11:00:33.667 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.676+0000 7fa2f37fe640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fa2f409c8c0 0x7fa2f414d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.137:46898/0 (socket says 172.21.15.137:46898) 2024-03-29T11:00:33.667 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.676+0000 7fa2f37fe640 1 -- 172.21.15.137:0/3160765756 learned_addr learned my addr 172.21.15.137:0/3160765756 (peer_addr_for_me v1:172.21.15.137:0/0) 2024-03-29T11:00:33.667 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 1 -- 172.21.15.137:0/3160765756 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2492250661 0 0) 0x7fa2f409b960 con 0x7fa2f409c8c0 2024-03-29T11:00:33.667 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.667 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.667 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.667 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:00:33.667 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 1 -- 172.21.15.137:0/3160765756 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2dc003520 con 0x7fa2f409c8c0 2024-03-29T11:00:33.667 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 1 -- 172.21.15.137:0/3160765756 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1817393680 0 0) 0x7fa2f40b2ec0 con 0x7fa2f40b8240 2024-03-29T11:00:33.668 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.668 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.668 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.668 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:00:33.668 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 1 -- 172.21.15.137:0/3160765756 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2f409b960 con 0x7fa2f40b8240 2024-03-29T11:00:33.668 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 1 -- 172.21.15.137:0/3160765756 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700314917 0 0) 0x7fa2f409bd80 con 0x7fa2f40b1f60 2024-03-29T11:00:33.668 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.668 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.668 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.668 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:00:33.668 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 1 -- 172.21.15.137:0/3160765756 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2f40b2ec0 con 0x7fa2f40b1f60 2024-03-29T11:00:33.668 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 1 -- 172.21.15.137:0/3160765756 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 30389154 0 0) 0x7fa2dc003520 con 0x7fa2f409c8c0 2024-03-29T11:00:33.668 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 1 -- 172.21.15.137:0/3160765756 >> v1:172.21.15.71:6790/0 conn(0x7fa2f40b8240 legacy=0x7fa2f413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.668 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 1 -- 172.21.15.137:0/3160765756 >> v1:172.21.15.71:6789/0 conn(0x7fa2f40b1f60 legacy=0x7fa2f409c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.668 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:00:33.669 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 1 monclient: found mon.noname-c 2024-03-29T11:00:33.669 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:33.669 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.137:6789/0 2024-03-29T11:00:33.669 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 1 -- 172.21.15.137:0/3160765756 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f413fb60 con 0x7fa2f409c8c0 2024-03-29T11:00:33.669 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:00:33.669 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:00:33.669 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:33.669 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f978d640 5 monclient: authenticate success, global_id 4132 2024-03-29T11:00:33.669 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f978d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:33.669 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 1 -- 172.21.15.137:0/3160765756 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fa2ec001a10 con 0x7fa2f409c8c0 2024-03-29T11:00:33.669 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.669 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:33.669 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 10 monclient: dump: 2024-03-29T11:00:33.669 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.669 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.670 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.670 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.670 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.670 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.670 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.670 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.670 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.670 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.670 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f978d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:33.670 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 1 -- 172.21.15.137:0/3160765756 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2ec001bf0 con 0x7fa2f409c8c0 2024-03-29T11:00:33.670 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:33.670 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 1 -- 172.21.15.137:0/3160765756 <== mon.1 v1:172.21.15.137:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fa2ec002000 con 0x7fa2f409c8c0 2024-03-29T11:00:33.670 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f978d640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:33.670 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f978d640 4 set_mon_vals no callback set 2024-03-29T11:00:33.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:33.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.677+0000 7fa2f1ffb640 10 monclient: dump: 2024-03-29T11:00:33.671 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.671 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.671 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.671 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.671 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.671 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.671 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.671 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.671 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.671 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.678+0000 7fa2f978d640 10 monclient: shutdown 2024-03-29T11:00:33.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.678+0000 7fa2f978d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:33.672 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.678+0000 7fa2f978d640 1 -- 172.21.15.137:0/3160765756 >> v1:172.21.15.137:6789/0 conn(0x7fa2f409c8c0 legacy=0x7fa2f414d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.672 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.678+0000 7fa2f978d640 1 -- 172.21.15.137:0/3160765756 shutdown_connections 2024-03-29T11:00:33.672 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.678+0000 7fa2f978d640 1 -- 172.21.15.137:0/3160765756 >> 172.21.15.137:0/3160765756 conn(0x7fa2f4066b00 msgr2=0x7fa2f4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:33.672 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.678+0000 7fa2f978d640 1 -- 172.21.15.137:0/3160765756 shutdown_connections 2024-03-29T11:00:33.672 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.678+0000 7fa2f978d640 1 -- 172.21.15.137:0/3160765756 wait complete. 2024-03-29T11:00:33.672 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.678+0000 7fa2f978d640 10 monclient: build_initial_monmap 2024-03-29T11:00:33.672 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.678+0000 7fa2f978d640 10 monclient: monmap: 2024-03-29T11:00:33.672 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 0 2024-03-29T11:00:33.672 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:33.672 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 0.000000 2024-03-29T11:00:33.672 INFO:teuthology.orchestra.run.smithi137.stderr:created 0.000000 2024-03-29T11:00:33.672 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:33.673 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.673 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:33.673 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:00:33.673 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:00:33.673 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.673 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.678+0000 7fa2f978d640 1 librados: starting msgr at 2024-03-29T11:00:33.673 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.678+0000 7fa2f978d640 1 librados: starting objecter 2024-03-29T11:00:33.673 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.679+0000 7fa2f978d640 1 Processor -- start 2024-03-29T11:00:33.673 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.679+0000 7fa2f978d640 1 -- start start 2024-03-29T11:00:33.673 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.679+0000 7fa2f978d640 1 librados: setting wanted keys 2024-03-29T11:00:33.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.679+0000 7fa2f978d640 1 librados: calling monclient init 2024-03-29T11:00:33.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.679+0000 7fa2f978d640 10 monclient: init 2024-03-29T11:00:33.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.679+0000 7fa2f978d640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:33.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.679+0000 7fa2f978d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:00:33.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.679+0000 7fa2f978d640 10 monclient(hunting): picked mon.noname-a con 0x7fa2f409c8c0 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:33.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.679+0000 7fa2f978d640 10 monclient(hunting): picked mon.noname-c con 0x7fa2f40b8240 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:33.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.679+0000 7fa2f978d640 10 monclient(hunting): picked mon.noname-b con 0x7fa2f4132b90 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:33.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.679+0000 7fa2f978d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f4135740 con 0x7fa2f409c8c0 2024-03-29T11:00:33.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.679+0000 7fa2f978d640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f40ad580 con 0x7fa2f4132b90 2024-03-29T11:00:33.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.679+0000 7fa2f978d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f40adad0 con 0x7fa2f40b8240 2024-03-29T11:00:33.675 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.679+0000 7fa2f978d640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:33.675 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.679+0000 7fa2f978d640 10 monclient(hunting): authenticate will time out at 779.013890s 2024-03-29T11:00:33.675 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.679+0000 7fa2f37fe640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fa2f4132b90 0x7fa2f41e2110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.137:46914/0 (socket says 172.21.15.137:46914) 2024-03-29T11:00:33.675 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.679+0000 7fa2f37fe640 1 -- 172.21.15.137:0/239074955 learned_addr learned my addr 172.21.15.137:0/239074955 (peer_addr_for_me v1:172.21.15.137:0/0) 2024-03-29T11:00:33.675 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 client.?.objecter ms_handle_connect 0x7fa2f4132b90 2024-03-29T11:00:33.675 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:33.675 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 client.?.objecter ms_handle_connect 0x7fa2f409c8c0 2024-03-29T11:00:33.675 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:33.675 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 client.?.objecter ms_handle_connect 0x7fa2f40b8240 2024-03-29T11:00:33.675 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:33.675 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 1 -- 172.21.15.137:0/239074955 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1223057134 0 0) 0x7fa2f40ad580 con 0x7fa2f4132b90 2024-03-29T11:00:33.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:00:33.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 1 -- 172.21.15.137:0/239074955 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2e0004eb0 con 0x7fa2f4132b90 2024-03-29T11:00:33.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 1 -- 172.21.15.137:0/239074955 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164965486 0 0) 0x7fa2f4135740 con 0x7fa2f409c8c0 2024-03-29T11:00:33.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:00:33.676 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 1 -- 172.21.15.137:0/239074955 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2f40ad580 con 0x7fa2f409c8c0 2024-03-29T11:00:33.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 1 -- 172.21.15.137:0/239074955 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069425808 0 0) 0x7fa2f40adad0 con 0x7fa2f40b8240 2024-03-29T11:00:33.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:00:33.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 1 -- 172.21.15.137:0/239074955 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2f4135740 con 0x7fa2f40b8240 2024-03-29T11:00:33.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 1 -- 172.21.15.137:0/239074955 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 817452092 0 0) 0x7fa2f40ad580 con 0x7fa2f409c8c0 2024-03-29T11:00:33.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 1 -- 172.21.15.137:0/239074955 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2f40adad0 con 0x7fa2f409c8c0 2024-03-29T11:00:33.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 1 -- 172.21.15.137:0/239074955 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fa2e4001970 con 0x7fa2f409c8c0 2024-03-29T11:00:33.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.677 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:00:33.678 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient(hunting): dump: 2024-03-29T11:00:33.678 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.678 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.678 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.678 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.678 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.678 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.678 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.679 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.679 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.679 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.679 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:33.679 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 1 -- 172.21.15.137:0/239074955 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3111121339 0 0) 0x7fa2e0004eb0 con 0x7fa2f4132b90 2024-03-29T11:00:33.679 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 1 -- 172.21.15.137:0/239074955 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2f40ad580 con 0x7fa2f4132b90 2024-03-29T11:00:33.679 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 1 -- 172.21.15.137:0/239074955 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2172988773 0 0) 0x7fa2f4135740 con 0x7fa2f40b8240 2024-03-29T11:00:33.679 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 1 -- 172.21.15.137:0/239074955 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2e0004eb0 con 0x7fa2f40b8240 2024-03-29T11:00:33.679 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 1 -- 172.21.15.137:0/239074955 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fa2ec001f00 con 0x7fa2f4132b90 2024-03-29T11:00:33.679 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.679 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:33.679 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient(hunting): dump: 2024-03-29T11:00:33.679 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.680 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.680 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.680 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.680 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.680 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.680 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.680 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.680 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.680 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.680 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 1 -- 172.21.15.137:0/239074955 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fa2e80017f0 con 0x7fa2f40b8240 2024-03-29T11:00:33.680 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.681 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:33.681 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient(hunting): dump: 2024-03-29T11:00:33.681 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.681 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.681 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.681 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.681 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.681 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.681 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.681 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.681 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.681 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.682 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 1 -- 172.21.15.137:0/239074955 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 302056366 0 0) 0x7fa2f40adad0 con 0x7fa2f409c8c0 2024-03-29T11:00:33.682 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 1 -- 172.21.15.137:0/239074955 >> v1:172.21.15.71:6790/0 conn(0x7fa2f40b8240 legacy=0x7fa2f40adef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.682 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 1 -- 172.21.15.137:0/239074955 >> v1:172.21.15.137:6789/0 conn(0x7fa2f4132b90 legacy=0x7fa2f41e2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.682 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:00:33.682 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 1 monclient: found mon.a 2024-03-29T11:00:33.682 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:33.682 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:00:33.682 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 1 -- 172.21.15.137:0/239074955 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f40ad7f0 con 0x7fa2f409c8c0 2024-03-29T11:00:33.682 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient: _finish_auth 0 2024-03-29T11:00:33.682 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:00:33.682 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:33.683 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2f978d640 5 monclient: authenticate success, global_id 4140 2024-03-29T11:00:33.683 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2f978d640 10 monclient: _renew_subs 2024-03-29T11:00:33.683 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2f978d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:00:33.683 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2f978d640 1 -- 172.21.15.137:0/239074955 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f4135490 con 0x7fa2f409c8c0 2024-03-29T11:00:33.683 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2f978d640 10 client.4140.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:33.683 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2f978d640 10 monclient: _renew_subs 2024-03-29T11:00:33.683 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2f978d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:00:33.683 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2f978d640 1 -- 172.21.15.137:0/239074955 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f4134920 con 0x7fa2f409c8c0 2024-03-29T11:00:33.683 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2f978d640 1 librados: init done 2024-03-29T11:00:33.683 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 1 -- 172.21.15.137:0/239074955 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2e40035f0 con 0x7fa2f409c8c0 2024-03-29T11:00:33.684 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:33.684 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 1 -- 172.21.15.137:0/239074955 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fa2e4003900 con 0x7fa2f409c8c0 2024-03-29T11:00:33.684 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.684 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:33.684 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 monclient: dump: 2024-03-29T11:00:33.684 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.684 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.684 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.684 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.684 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.684 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.684 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.685 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.685 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.685 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.685 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2f17fa640 4 set_mon_vals no callback set 2024-03-29T11:00:33.685 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 1 -- 172.21.15.137:0/239074955 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa2e4003df0 con 0x7fa2f409c8c0 2024-03-29T11:00:33.685 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 1 -- 172.21.15.137:0/239074955 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2218036860 0 0) 0x7fa2e4005590 con 0x7fa2f409c8c0 2024-03-29T11:00:33.685 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 10 client.4140.objecter ms_dispatch 0x7fa2f4000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:00:33.685 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 3 client.4140.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:00:33.685 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 3 client.4140.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:00:33.685 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.680+0000 7fa2db7fe640 20 client.4140.objecter dump_active .. 0 homeless 2024-03-29T11:00:33.685 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.681+0000 7fa2f978d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:33.686 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.681+0000 7fa2f978d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:33.686 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.681+0000 7fa2f978d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:00:33.686 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.681+0000 7fa2f978d640 1 -- 172.21.15.137:0/239074955 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2d4005080 con 0x7fa2f409c8c0 2024-03-29T11:00:33.686 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.683+0000 7fa2db7fe640 1 -- 172.21.15.137:0/239074955 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa2e4005130 con 0x7fa2f409c8c0 2024-03-29T11:00:33.686 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.683+0000 7fa2db7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:33.686 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.683+0000 7fa2db7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:33.687 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T11:00:33.702+0000 7fab49e36040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:00:33.687 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:00:33.697+0000 7f5976230040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:00:33.730 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:00:33.749+0000 7f5976230040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:00:33.735 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T11:00:33.755+0000 7fab49e36040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:00:33.736 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.756+0000 7fa2f978d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a9434f80-2148-42b6-a4d9-ca9b055b61c9", "id": 4}] 2024-03-29T11:00:33.736 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.756+0000 7fa2f978d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a9434f80-2148-42b6-a4d9-ca9b055b61c9", "id": 4}] 2024-03-29T11:00:33.736 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.756+0000 7fa2f978d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:00:33.736 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.756+0000 7fa2f978d640 1 -- 172.21.15.137:0/239074955 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a9434f80-2148-42b6-a4d9-ca9b055b61c9", "id": 4} v 0) v1 -- 0x7fa2d40058b0 con 0x7fa2f409c8c0 2024-03-29T11:00:33.737 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.758+0000 7fa2db7fe640 1 -- 172.21.15.137:0/239074955 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a9434f80-2148-42b6-a4d9-ca9b055b61c9", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 4060269538 0 1697012583) 0x7fa2e40057c0 con 0x7fa2f409c8c0 2024-03-29T11:00:33.738 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.758+0000 7fa2db7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a9434f80-2148-42b6-a4d9-ca9b055b61c9", "id": 4}] 2024-03-29T11:00:33.738 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.758+0000 7fa2db7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:33.738 INFO:teuthology.orchestra.run.smithi137.stdout:4 2024-03-29T11:00:33.739 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.758+0000 7fa2f978d640 10 librados: watch_flush enter 2024-03-29T11:00:33.740 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.758+0000 7fa2f978d640 10 librados: watch_flush exit 2024-03-29T11:00:33.740 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.758+0000 7fa2f978d640 20 client.4140.objecter shutdown clearing up homeless session... 2024-03-29T11:00:33.740 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.758+0000 7fa2f978d640 10 client.4140.objecter successfully canceled tick 2024-03-29T11:00:33.740 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.758+0000 7fa2f978d640 10 monclient: shutdown 2024-03-29T11:00:33.740 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.758+0000 7fa2f978d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:33.740 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.758+0000 7fa2f978d640 1 -- 172.21.15.137:0/239074955 >> v1:172.21.15.71:6789/0 conn(0x7fa2f409c8c0 legacy=0x7fa2f4134d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.740 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.758+0000 7fa2f978d640 1 -- 172.21.15.137:0/239074955 shutdown_connections 2024-03-29T11:00:33.740 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.758+0000 7fa2f978d640 1 -- 172.21.15.137:0/239074955 >> 172.21.15.137:0/239074955 conn(0x7fa2f4066b00 msgr2=0x7fa2f406bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:33.740 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.759+0000 7fa2f978d640 1 -- 172.21.15.137:0/239074955 shutdown_connections 2024-03-29T11:00:33.740 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.759+0000 7fa2f978d640 1 -- 172.21.15.137:0/239074955 wait complete. 2024-03-29T11:00:33.740 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.759+0000 7fa2f978d640 1 librados: shutdown 2024-03-29T11:00:33.749 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new 21446639-1b33-4ae5-9997-6fdfad0dc87e 5 2024-03-29T11:00:33.778 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:00:33.797+0000 7f5976230040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:00:33.785 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T11:00:33.805+0000 7fab49e36040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:00:33.885 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.905+0000 7fca52e4a640 10 monclient: get_monmap_and_config 2024-03-29T11:00:33.885 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.905+0000 7fca52e4a640 10 monclient: build_initial_monmap 2024-03-29T11:00:33.885 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.905+0000 7fca52e4a640 10 monclient: monmap: 2024-03-29T11:00:33.886 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 0 2024-03-29T11:00:33.886 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.886 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T11:00:33.906673+0000 2024-03-29T11:00:33.886 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T11:00:33.906673+0000 2024-03-29T11:00:33.886 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:33.886 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.886 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:33.886 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:00:33.886 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:00:33.886 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.898 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.906+0000 7fca52e4a640 1 Processor -- start 2024-03-29T11:00:33.898 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.906+0000 7fca52e4a640 1 -- start start 2024-03-29T11:00:33.898 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.906+0000 7fca52e4a640 10 monclient: init 2024-03-29T11:00:33.898 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.906+0000 7fca52e4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:33.898 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.906+0000 7fca52e4a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:00:33.899 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.906+0000 7fca52e4a640 10 monclient(hunting): picked mon.noname-b con 0x7fca4c09ac40 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:33.899 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.906+0000 7fca52e4a640 10 monclient(hunting): picked mon.noname-a con 0x7fca4c09c870 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:33.899 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.906+0000 7fca52e4a640 10 monclient(hunting): picked mon.noname-c con 0x7fca4c0b4360 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:33.899 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.906+0000 7fca52e4a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca4c09dec0 con 0x7fca4c09c870 2024-03-29T11:00:33.899 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.906+0000 7fca52e4a640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca4c09da30 con 0x7fca4c0b4360 2024-03-29T11:00:33.899 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca52e4a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca4c0b39c0 con 0x7fca4c09ac40 2024-03-29T11:00:33.899 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca52e4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:33.899 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca52e4a640 10 monclient(hunting): authenticate will time out at 779.241168s 2024-03-29T11:00:33.899 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4bfff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fca4c09c870 0x7fca4c13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.137:47820/0 (socket says 172.21.15.137:47820) 2024-03-29T11:00:33.899 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca513e7640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fca4c0b4360 0x7fca4c14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.137:46924/0 (socket says 172.21.15.137:46924) 2024-03-29T11:00:33.899 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca50be6640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fca4c09ac40 0x7fca4c0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.137:44036/0 (socket says 172.21.15.137:44036) 2024-03-29T11:00:33.900 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4bfff640 1 -- 172.21.15.137:0/2141312477 learned_addr learned my addr 172.21.15.137:0/2141312477 (peer_addr_for_me v1:172.21.15.137:0/0) 2024-03-29T11:00:33.900 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 1 -- 172.21.15.137:0/2141312477 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851454077 0 0) 0x7fca4c09da30 con 0x7fca4c0b4360 2024-03-29T11:00:33.900 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.900 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.900 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.900 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:00:33.900 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 1 -- 172.21.15.137:0/2141312477 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca3c003520 con 0x7fca4c0b4360 2024-03-29T11:00:33.900 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 1 -- 172.21.15.137:0/2141312477 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745404208 0 0) 0x7fca38001040 con 0x7fca4c09c870 2024-03-29T11:00:33.900 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.900 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.900 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.900 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:00:33.900 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 1 -- 172.21.15.137:0/2141312477 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca4c09da30 con 0x7fca4c09c870 2024-03-29T11:00:33.900 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 1 -- 172.21.15.137:0/2141312477 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2446536918 0 0) 0x7fca4c0b39c0 con 0x7fca4c09ac40 2024-03-29T11:00:33.901 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.901 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.901 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.901 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:00:33.901 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 1 -- 172.21.15.137:0/2141312477 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca38001040 con 0x7fca4c09ac40 2024-03-29T11:00:33.901 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 1 -- 172.21.15.137:0/2141312477 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1394140451 0 0) 0x7fca3c003520 con 0x7fca4c0b4360 2024-03-29T11:00:33.901 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 1 -- 172.21.15.137:0/2141312477 >> v1:172.21.15.71:6790/0 conn(0x7fca4c09ac40 legacy=0x7fca4c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.901 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 1 -- 172.21.15.137:0/2141312477 >> v1:172.21.15.71:6789/0 conn(0x7fca4c09c870 legacy=0x7fca4c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.901 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:00:33.901 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 1 monclient: found mon.noname-c 2024-03-29T11:00:33.901 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:33.901 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.137:6789/0 2024-03-29T11:00:33.901 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 1 -- 172.21.15.137:0/2141312477 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca4c0b5160 con 0x7fca4c0b4360 2024-03-29T11:00:33.901 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:00:33.901 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:00:33.902 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:33.902 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca52e4a640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:00:33.902 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 1 -- 172.21.15.137:0/2141312477 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fca44001a10 con 0x7fca4c0b4360 2024-03-29T11:00:33.902 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca52e4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:33.902 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.902 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:33.902 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 10 monclient: dump: 2024-03-29T11:00:33.902 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.902 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.902 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.902 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.902 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.902 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.902 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.903 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.903 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.903 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.903 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca52e4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:33.903 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 1 -- 172.21.15.137:0/2141312477 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca44001bf0 con 0x7fca4c0b4360 2024-03-29T11:00:33.903 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:33.903 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 1 -- 172.21.15.137:0/2141312477 <== mon.1 v1:172.21.15.137:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fca44002040 con 0x7fca4c0b4360 2024-03-29T11:00:33.903 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca52e4a640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:33.903 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca52e4a640 4 set_mon_vals no callback set 2024-03-29T11:00:33.904 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.904 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:33.904 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.907+0000 7fca4b7fe640 10 monclient: dump: 2024-03-29T11:00:33.904 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.904 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.904 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.904 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.904 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.905 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.905 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.905 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.905 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.905 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.905 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.908+0000 7fca52e4a640 10 monclient: shutdown 2024-03-29T11:00:33.905 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.908+0000 7fca52e4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:33.905 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.908+0000 7fca52e4a640 1 -- 172.21.15.137:0/2141312477 >> v1:172.21.15.137:6789/0 conn(0x7fca4c0b4360 legacy=0x7fca4c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.905 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.908+0000 7fca52e4a640 1 -- 172.21.15.137:0/2141312477 shutdown_connections 2024-03-29T11:00:33.905 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.908+0000 7fca52e4a640 1 -- 172.21.15.137:0/2141312477 >> 172.21.15.137:0/2141312477 conn(0x7fca4c066b00 msgr2=0x7fca4c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:33.905 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.908+0000 7fca52e4a640 1 -- 172.21.15.137:0/2141312477 shutdown_connections 2024-03-29T11:00:33.905 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.908+0000 7fca52e4a640 1 -- 172.21.15.137:0/2141312477 wait complete. 2024-03-29T11:00:33.906 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.909+0000 7fca52e4a640 10 monclient: build_initial_monmap 2024-03-29T11:00:33.906 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.909+0000 7fca52e4a640 10 monclient: monmap: 2024-03-29T11:00:33.906 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 0 2024-03-29T11:00:33.906 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:33.906 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 0.000000 2024-03-29T11:00:33.906 INFO:teuthology.orchestra.run.smithi137.stderr:created 0.000000 2024-03-29T11:00:33.906 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:33.906 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.906 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:33.906 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:00:33.906 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:00:33.906 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.906 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.909+0000 7fca52e4a640 1 librados: starting msgr at 2024-03-29T11:00:33.906 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.909+0000 7fca52e4a640 1 librados: starting objecter 2024-03-29T11:00:33.907 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.909+0000 7fca52e4a640 1 Processor -- start 2024-03-29T11:00:33.907 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.909+0000 7fca52e4a640 1 -- start start 2024-03-29T11:00:33.907 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.909+0000 7fca52e4a640 1 librados: setting wanted keys 2024-03-29T11:00:33.907 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.909+0000 7fca52e4a640 1 librados: calling monclient init 2024-03-29T11:00:33.907 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.909+0000 7fca52e4a640 10 monclient: init 2024-03-29T11:00:33.907 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca52e4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:33.907 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca52e4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:00:33.907 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca52e4a640 10 monclient(hunting): picked mon.noname-a con 0x7fca4c09ac40 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:33.907 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca52e4a640 10 monclient(hunting): picked mon.noname-b con 0x7fca4c146190 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:33.907 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca52e4a640 10 monclient(hunting): picked mon.noname-c con 0x7fca4c1d7980 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:33.907 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca52e4a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca4c13ffa0 con 0x7fca4c09ac40 2024-03-29T11:00:33.907 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca52e4a640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca4c1d7df0 con 0x7fca4c146190 2024-03-29T11:00:33.907 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca52e4a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca4c0b1620 con 0x7fca4c1d7980 2024-03-29T11:00:33.908 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca52e4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:33.908 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca52e4a640 10 monclient(hunting): authenticate will time out at 779.244144s 2024-03-29T11:00:33.908 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca4bfff640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fca4c146190 0x7fca4c1d7270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.137:46938/0 (socket says 172.21.15.137:46938) 2024-03-29T11:00:33.908 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca4bfff640 1 -- 172.21.15.137:0/1749323906 learned_addr learned my addr 172.21.15.137:0/1749323906 (peer_addr_for_me v1:172.21.15.137:0/0) 2024-03-29T11:00:33.908 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 client.?.objecter ms_handle_connect 0x7fca4c146190 2024-03-29T11:00:33.908 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:33.908 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 client.?.objecter ms_handle_connect 0x7fca4c09ac40 2024-03-29T11:00:33.908 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:33.908 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 client.?.objecter ms_handle_connect 0x7fca4c1d7980 2024-03-29T11:00:33.908 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:33.908 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 1 -- 172.21.15.137:0/1749323906 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159584048 0 0) 0x7fca4c1d7df0 con 0x7fca4c146190 2024-03-29T11:00:33.908 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.909 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.909 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.909 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:00:33.909 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 1 -- 172.21.15.137:0/1749323906 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca30004eb0 con 0x7fca4c146190 2024-03-29T11:00:33.909 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 1 -- 172.21.15.137:0/1749323906 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1901085880 0 0) 0x7fca4c13ffa0 con 0x7fca4c09ac40 2024-03-29T11:00:33.909 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.909 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.909 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.909 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:00:33.909 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 1 -- 172.21.15.137:0/1749323906 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca4c1d7df0 con 0x7fca4c09ac40 2024-03-29T11:00:33.909 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 1 -- 172.21.15.137:0/1749323906 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1966183266 0 0) 0x7fca4c0b1620 con 0x7fca4c1d7980 2024-03-29T11:00:33.909 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.909 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.909 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.909 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:00:33.910 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 1 -- 172.21.15.137:0/1749323906 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca4c13ffa0 con 0x7fca4c1d7980 2024-03-29T11:00:33.910 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 1 -- 172.21.15.137:0/1749323906 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2062385199 0 0) 0x7fca30004eb0 con 0x7fca4c146190 2024-03-29T11:00:33.910 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 1 -- 172.21.15.137:0/1749323906 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca4c0b1620 con 0x7fca4c146190 2024-03-29T11:00:33.910 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 1 -- 172.21.15.137:0/1749323906 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fca38001450 con 0x7fca4c146190 2024-03-29T11:00:33.910 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.910 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:33.910 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 monclient(hunting): dump: 2024-03-29T11:00:33.910 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.910 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.910 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.910 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.910 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.911 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.911 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.911 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.911 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.911 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.911 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:33.911 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 1 -- 172.21.15.137:0/1749323906 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2930860820 0 0) 0x7fca4c1d7df0 con 0x7fca4c09ac40 2024-03-29T11:00:33.911 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 1 -- 172.21.15.137:0/1749323906 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca30004eb0 con 0x7fca4c09ac40 2024-03-29T11:00:33.911 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 1 -- 172.21.15.137:0/1749323906 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fca400018b0 con 0x7fca4c09ac40 2024-03-29T11:00:33.911 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.911 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:33.911 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 monclient(hunting): dump: 2024-03-29T11:00:33.911 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.911 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.911 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.912 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.912 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.912 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.912 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.912 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.912 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.912 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.912 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 1 -- 172.21.15.137:0/1749323906 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 988542970 0 0) 0x7fca4c13ffa0 con 0x7fca4c1d7980 2024-03-29T11:00:33.912 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 1 -- 172.21.15.137:0/1749323906 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca4c1d7df0 con 0x7fca4c1d7980 2024-03-29T11:00:33.912 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 1 -- 172.21.15.137:0/1749323906 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fca44001d60 con 0x7fca4c1d7980 2024-03-29T11:00:33.912 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.912 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:33.912 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 monclient(hunting): dump: 2024-03-29T11:00:33.912 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.913 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.913 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.913 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.913 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.913 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.913 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.913 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.913 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.913 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.913 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 1 -- 172.21.15.137:0/1749323906 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3955062661 0 0) 0x7fca4c0b1620 con 0x7fca4c146190 2024-03-29T11:00:33.913 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 1 -- 172.21.15.137:0/1749323906 >> v1:172.21.15.71:6790/0 conn(0x7fca4c1d7980 legacy=0x7fca4c1e1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.913 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 1 -- 172.21.15.137:0/1749323906 >> v1:172.21.15.71:6789/0 conn(0x7fca4c09ac40 legacy=0x7fca4c142a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.913 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 monclient: _finish_hunting 0 2024-03-29T11:00:33.913 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 1 monclient: found mon.b 2024-03-29T11:00:33.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:33.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:33.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 1 -- 172.21.15.137:0/1749323906 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca4c143160 con 0x7fca4c146190 2024-03-29T11:00:33.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 monclient: _finish_auth 0 2024-03-29T11:00:33.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 10 monclient: _check_auth_tickets 2024-03-29T11:00:33.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:33.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca52e4a640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:00:33.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca52e4a640 10 monclient: _renew_subs 2024-03-29T11:00:33.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca52e4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:33.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.910+0000 7fca52e4a640 1 -- 172.21.15.137:0/1749323906 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca4c0b12b0 con 0x7fca4c146190 2024-03-29T11:00:33.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.911+0000 7fca52e4a640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:33.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.911+0000 7fca52e4a640 10 monclient: _renew_subs 2024-03-29T11:00:33.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.911+0000 7fca52e4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:33.914 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.911+0000 7fca52e4a640 1 -- 172.21.15.137:0/1749323906 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca4c1e3260 con 0x7fca4c146190 2024-03-29T11:00:33.915 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.911+0000 7fca52e4a640 1 librados: init done 2024-03-29T11:00:33.915 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.911+0000 7fca497fa640 1 -- 172.21.15.137:0/1749323906 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca38004a90 con 0x7fca4c146190 2024-03-29T11:00:33.915 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.911+0000 7fca497fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:33.915 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.911+0000 7fca497fa640 1 -- 172.21.15.137:0/1749323906 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fca38004da0 con 0x7fca4c146190 2024-03-29T11:00:33.915 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.911+0000 7fca497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.915 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.911+0000 7fca497fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:33.915 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.911+0000 7fca497fa640 10 monclient: dump: 2024-03-29T11:00:33.915 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:33.915 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:33.915 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.915 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:33.915 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.915 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:33.915 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:33.915 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:33.916 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:33.916 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:33.916 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.911+0000 7fca4affd640 4 set_mon_vals no callback set 2024-03-29T11:00:33.916 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.911+0000 7fca497fa640 1 -- 172.21.15.137:0/1749323906 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fca38005290 con 0x7fca4c146190 2024-03-29T11:00:33.916 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.911+0000 7fca497fa640 1 -- 172.21.15.137:0/1749323906 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 3524909980 0 0) 0x7fca380069c0 con 0x7fca4c146190 2024-03-29T11:00:33.916 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.911+0000 7fca497fa640 10 client.4141.objecter ms_dispatch 0x7fca4c000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:00:33.916 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.911+0000 7fca497fa640 3 client.4141.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:00:33.916 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.911+0000 7fca497fa640 3 client.4141.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:00:33.916 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.911+0000 7fca497fa640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-29T11:00:33.916 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.911+0000 7fca52e4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:33.916 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.911+0000 7fca52e4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:33.916 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.911+0000 7fca52e4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:33.916 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.911+0000 7fca52e4a640 1 -- 172.21.15.137:0/1749323906 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca4c09b020 con 0x7fca4c146190 2024-03-29T11:00:33.916 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.913+0000 7fca497fa640 1 -- 172.21.15.137:0/1749323906 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fca38006440 con 0x7fca4c146190 2024-03-29T11:00:33.917 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.913+0000 7fca497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:33.917 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.913+0000 7fca497fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:33.966 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.986+0000 7fca52e4a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "21446639-1b33-4ae5-9997-6fdfad0dc87e", "id": 5}] 2024-03-29T11:00:33.966 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.986+0000 7fca52e4a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "21446639-1b33-4ae5-9997-6fdfad0dc87e", "id": 5}] 2024-03-29T11:00:33.967 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.986+0000 7fca52e4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:33.967 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.986+0000 7fca52e4a640 1 -- 172.21.15.137:0/1749323906 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd new", "uuid": "21446639-1b33-4ae5-9997-6fdfad0dc87e", "id": 5} v 0) v1 -- 0x7fca4c04f730 con 0x7fca4c146190 2024-03-29T11:00:33.970 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.990+0000 7fca497fa640 1 -- 172.21.15.137:0/1749323906 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "21446639-1b33-4ae5-9997-6fdfad0dc87e", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 499470163 0 1988412176) 0x7fca38006790 con 0x7fca4c146190 2024-03-29T11:00:33.970 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.990+0000 7fca497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "21446639-1b33-4ae5-9997-6fdfad0dc87e", "id": 5}] 2024-03-29T11:00:33.970 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.990+0000 7fca497fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:33.971 INFO:teuthology.orchestra.run.smithi137.stdout:5 2024-03-29T11:00:33.972 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.990+0000 7fca52e4a640 10 librados: watch_flush enter 2024-03-29T11:00:33.972 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.990+0000 7fca52e4a640 10 librados: watch_flush exit 2024-03-29T11:00:33.972 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.990+0000 7fca52e4a640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-29T11:00:33.972 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.990+0000 7fca52e4a640 10 client.4141.objecter successfully canceled tick 2024-03-29T11:00:33.972 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.991+0000 7fca52e4a640 10 monclient: shutdown 2024-03-29T11:00:33.972 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.991+0000 7fca52e4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:33.973 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.991+0000 7fca52e4a640 1 -- 172.21.15.137:0/1749323906 >> v1:172.21.15.137:6789/0 conn(0x7fca4c146190 legacy=0x7fca4c1d7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.973 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.991+0000 7fca52e4a640 1 -- 172.21.15.137:0/1749323906 shutdown_connections 2024-03-29T11:00:33.973 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.991+0000 7fca52e4a640 1 -- 172.21.15.137:0/1749323906 >> 172.21.15.137:0/1749323906 conn(0x7fca4c066b00 msgr2=0x7fca4c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:33.973 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.991+0000 7fca52e4a640 1 -- 172.21.15.137:0/1749323906 shutdown_connections 2024-03-29T11:00:33.973 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.991+0000 7fca52e4a640 1 -- 172.21.15.137:0/1749323906 wait complete. 2024-03-29T11:00:33.973 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:33.991+0000 7fca52e4a640 1 librados: shutdown 2024-03-29T11:00:33.981 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new aa3fdfbe-4e06-411f-9b6e-88732ca48f90 6 2024-03-29T11:00:34.082 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.101+0000 7fbc74582640 10 monclient: get_monmap_and_config 2024-03-29T11:00:34.082 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.101+0000 7fbc74582640 10 monclient: build_initial_monmap 2024-03-29T11:00:34.082 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.101+0000 7fbc74582640 10 monclient: monmap: 2024-03-29T11:00:34.082 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 0 2024-03-29T11:00:34.082 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:34.082 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T11:00:34.102787+0000 2024-03-29T11:00:34.082 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T11:00:34.102787+0000 2024-03-29T11:00:34.082 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:34.082 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:34.083 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:34.083 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:00:34.083 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:00:34.083 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:34.083 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.102+0000 7fbc74582640 1 Processor -- start 2024-03-29T11:00:34.083 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.102+0000 7fbc74582640 1 -- start start 2024-03-29T11:00:34.083 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.102+0000 7fbc74582640 10 monclient: init 2024-03-29T11:00:34.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.102+0000 7fbc74582640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:34.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc74582640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:00:34.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc74582640 10 monclient(hunting): picked mon.noname-a con 0x7fbc6c09ac40 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:34.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc74582640 10 monclient(hunting): picked mon.noname-b con 0x7fbc6c09c870 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:34.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc74582640 10 monclient(hunting): picked mon.noname-c con 0x7fbc6c0b4360 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:34.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc74582640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc6c09dec0 con 0x7fbc6c09ac40 2024-03-29T11:00:34.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc74582640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc6c09da30 con 0x7fbc6c0b4360 2024-03-29T11:00:34.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc74582640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc6c0b39c0 con 0x7fbc6c09c870 2024-03-29T11:00:34.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc74582640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:34.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc74582640 10 monclient(hunting): authenticate will time out at 779.437138s 2024-03-29T11:00:34.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc72b1f640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fbc6c0b4360 0x7fbc6c14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.137:46942/0 (socket says 172.21.15.137:46942) 2024-03-29T11:00:34.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc72b1f640 1 -- 172.21.15.137:0/446342368 learned_addr learned my addr 172.21.15.137:0/446342368 (peer_addr_for_me v1:172.21.15.137:0/0) 2024-03-29T11:00:34.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 1 -- 172.21.15.137:0/446342368 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281370907 0 0) 0x7fbc6c09da30 con 0x7fbc6c0b4360 2024-03-29T11:00:34.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 10 monclient(hunting): _negotiate 2024-03-29T11:00:34.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:34.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:34.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:00:34.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 1 -- 172.21.15.137:0/446342368 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc58003520 con 0x7fbc6c0b4360 2024-03-29T11:00:34.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 1 -- 172.21.15.137:0/446342368 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6775344 0 0) 0x7fbc6c0b39c0 con 0x7fbc6c09c870 2024-03-29T11:00:34.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 10 monclient(hunting): _negotiate 2024-03-29T11:00:34.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:34.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:34.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:00:34.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 1 -- 172.21.15.137:0/446342368 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc6c09da30 con 0x7fbc6c09c870 2024-03-29T11:00:34.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 1 -- 172.21.15.137:0/446342368 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3841224001 0 0) 0x7fbc68001110 con 0x7fbc6c09ac40 2024-03-29T11:00:34.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 10 monclient(hunting): _negotiate 2024-03-29T11:00:34.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:34.094 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:34.094 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:00:34.094 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 1 -- 172.21.15.137:0/446342368 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc6c0b39c0 con 0x7fbc6c09ac40 2024-03-29T11:00:34.094 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 1 -- 172.21.15.137:0/446342368 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2154970698 0 0) 0x7fbc58003520 con 0x7fbc6c0b4360 2024-03-29T11:00:34.094 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 1 -- 172.21.15.137:0/446342368 >> v1:172.21.15.71:6790/0 conn(0x7fbc6c09c870 legacy=0x7fbc6c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:34.094 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 1 -- 172.21.15.137:0/446342368 >> v1:172.21.15.71:6789/0 conn(0x7fbc6c09ac40 legacy=0x7fbc6c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:34.094 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 10 monclient: _finish_hunting 0 2024-03-29T11:00:34.094 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 1 monclient: found mon.noname-c 2024-03-29T11:00:34.094 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:34.094 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.137:6789/0 2024-03-29T11:00:34.094 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 1 -- 172.21.15.137:0/446342368 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc6c0b5160 con 0x7fbc6c0b4360 2024-03-29T11:00:34.094 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 10 monclient: _finish_auth 0 2024-03-29T11:00:34.094 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 10 monclient: _check_auth_tickets 2024-03-29T11:00:34.094 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:34.094 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 1 -- 172.21.15.137:0/446342368 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fbc64001a10 con 0x7fbc6c0b4360 2024-03-29T11:00:34.095 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc74582640 5 monclient: authenticate success, global_id 4144 2024-03-29T11:00:34.095 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc74582640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:34.095 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:34.095 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:34.095 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 10 monclient: dump: 2024-03-29T11:00:34.095 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:34.095 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:34.095 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.095 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.095 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:34.095 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:34.095 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:34.095 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:34.095 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:34.095 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:34.096 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc74582640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:34.096 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 1 -- 172.21.15.137:0/446342368 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc64001bf0 con 0x7fbc6c0b4360 2024-03-29T11:00:34.096 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:34.096 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 1 -- 172.21.15.137:0/446342368 <== mon.1 v1:172.21.15.137:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fbc64002000 con 0x7fbc6c0b4360 2024-03-29T11:00:34.096 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc74582640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:34.096 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc74582640 4 set_mon_vals no callback set 2024-03-29T11:00:34.096 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:34.096 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:34.096 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.103+0000 7fbc7131c640 10 monclient: dump: 2024-03-29T11:00:34.096 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:34.096 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:34.096 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.096 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.096 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:34.096 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:34.097 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:34.097 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:34.097 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:34.097 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:34.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.104+0000 7fbc74582640 10 monclient: shutdown 2024-03-29T11:00:34.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.104+0000 7fbc74582640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:34.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.104+0000 7fbc74582640 1 -- 172.21.15.137:0/446342368 >> v1:172.21.15.137:6789/0 conn(0x7fbc6c0b4360 legacy=0x7fbc6c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:34.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.104+0000 7fbc74582640 1 -- 172.21.15.137:0/446342368 shutdown_connections 2024-03-29T11:00:34.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.104+0000 7fbc74582640 1 -- 172.21.15.137:0/446342368 >> 172.21.15.137:0/446342368 conn(0x7fbc6c066b00 msgr2=0x7fbc6c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:34.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.104+0000 7fbc74582640 1 -- 172.21.15.137:0/446342368 shutdown_connections 2024-03-29T11:00:34.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.104+0000 7fbc74582640 1 -- 172.21.15.137:0/446342368 wait complete. 2024-03-29T11:00:34.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.105+0000 7fbc74582640 10 monclient: build_initial_monmap 2024-03-29T11:00:34.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.105+0000 7fbc74582640 10 monclient: monmap: 2024-03-29T11:00:34.097 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 0 2024-03-29T11:00:34.098 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:34.098 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 0.000000 2024-03-29T11:00:34.098 INFO:teuthology.orchestra.run.smithi137.stderr:created 0.000000 2024-03-29T11:00:34.098 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:34.098 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:34.098 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:34.098 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:00:34.098 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:00:34.098 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:34.098 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.105+0000 7fbc74582640 1 librados: starting msgr at 2024-03-29T11:00:34.098 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.105+0000 7fbc74582640 1 librados: starting objecter 2024-03-29T11:00:34.098 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.105+0000 7fbc74582640 1 Processor -- start 2024-03-29T11:00:34.098 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.105+0000 7fbc74582640 1 -- start start 2024-03-29T11:00:34.098 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.105+0000 7fbc74582640 1 librados: setting wanted keys 2024-03-29T11:00:34.099 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.105+0000 7fbc74582640 1 librados: calling monclient init 2024-03-29T11:00:34.099 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.105+0000 7fbc74582640 10 monclient: init 2024-03-29T11:00:34.099 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.105+0000 7fbc74582640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:34.099 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.105+0000 7fbc74582640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:00:34.099 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.105+0000 7fbc74582640 10 monclient(hunting): picked mon.noname-a con 0x7fbc6c09ac40 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:34.099 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.105+0000 7fbc74582640 10 monclient(hunting): picked mon.noname-c con 0x7fbc6c1d7bd0 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:34.099 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.105+0000 7fbc74582640 10 monclient(hunting): picked mon.noname-b con 0x7fbc6c140600 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:34.099 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc74582640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc6c136aa0 con 0x7fbc6c09ac40 2024-03-29T11:00:34.099 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc74582640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc6c1423a0 con 0x7fbc6c140600 2024-03-29T11:00:34.099 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc74582640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc6c1d76d0 con 0x7fbc6c1d7bd0 2024-03-29T11:00:34.100 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc74582640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:34.100 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc74582640 10 monclient(hunting): authenticate will time out at 779.440044s 2024-03-29T11:00:34.100 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc72b1f640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fbc6c140600 0x7fbc6c140a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.137:46956/0 (socket says 172.21.15.137:46956) 2024-03-29T11:00:34.100 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc72b1f640 1 -- 172.21.15.137:0/581374179 learned_addr learned my addr 172.21.15.137:0/581374179 (peer_addr_for_me v1:172.21.15.137:0/0) 2024-03-29T11:00:34.100 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 client.?.objecter ms_handle_connect 0x7fbc6c140600 2024-03-29T11:00:34.100 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:34.100 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 client.?.objecter ms_handle_connect 0x7fbc6c1d7bd0 2024-03-29T11:00:34.100 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:34.100 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 client.?.objecter ms_handle_connect 0x7fbc6c09ac40 2024-03-29T11:00:34.100 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:34.100 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 1 -- 172.21.15.137:0/581374179 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4092473800 0 0) 0x7fbc6c1423a0 con 0x7fbc6c140600 2024-03-29T11:00:34.100 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:34.100 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:34.100 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:34.100 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient(hunting): my global_id is 4147 2024-03-29T11:00:34.101 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 1 -- 172.21.15.137:0/581374179 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc50004eb0 con 0x7fbc6c140600 2024-03-29T11:00:34.101 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 1 -- 172.21.15.137:0/581374179 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 81647580 0 0) 0x7fbc6c1d76d0 con 0x7fbc6c1d7bd0 2024-03-29T11:00:34.101 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:34.101 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:34.101 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:34.101 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:00:34.101 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 1 -- 172.21.15.137:0/581374179 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc6c1423a0 con 0x7fbc6c1d7bd0 2024-03-29T11:00:34.101 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 1 -- 172.21.15.137:0/581374179 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4221976253 0 0) 0x7fbc6c136aa0 con 0x7fbc6c09ac40 2024-03-29T11:00:34.101 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:34.101 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:34.102 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:34.102 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:00:34.102 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 1 -- 172.21.15.137:0/581374179 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc6c1d76d0 con 0x7fbc6c09ac40 2024-03-29T11:00:34.102 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 1 -- 172.21.15.137:0/581374179 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3001385100 0 0) 0x7fbc50004eb0 con 0x7fbc6c140600 2024-03-29T11:00:34.102 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 1 -- 172.21.15.137:0/581374179 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc6c136aa0 con 0x7fbc6c140600 2024-03-29T11:00:34.102 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 1 -- 172.21.15.137:0/581374179 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fbc64001f40 con 0x7fbc6c140600 2024-03-29T11:00:34.102 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:34.102 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:34.102 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient(hunting): dump: 2024-03-29T11:00:34.102 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:34.102 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:34.102 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.102 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.102 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:34.103 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:34.103 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:34.103 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:34.103 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:34.103 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:34.103 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:34.103 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 1 -- 172.21.15.137:0/581374179 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2929446378 0 0) 0x7fbc6c1423a0 con 0x7fbc6c1d7bd0 2024-03-29T11:00:34.103 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 1 -- 172.21.15.137:0/581374179 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc50004eb0 con 0x7fbc6c1d7bd0 2024-03-29T11:00:34.104 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 1 -- 172.21.15.137:0/581374179 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fbc5c001930 con 0x7fbc6c1d7bd0 2024-03-29T11:00:34.104 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:34.104 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:34.104 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient(hunting): dump: 2024-03-29T11:00:34.104 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:34.104 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:34.104 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.104 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.104 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:34.104 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:34.104 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:34.104 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:34.104 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:34.104 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:34.105 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 1 -- 172.21.15.137:0/581374179 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 67855116 0 0) 0x7fbc6c1d76d0 con 0x7fbc6c09ac40 2024-03-29T11:00:34.105 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 1 -- 172.21.15.137:0/581374179 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc6c1423a0 con 0x7fbc6c09ac40 2024-03-29T11:00:34.105 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 1 -- 172.21.15.137:0/581374179 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fbc68001860 con 0x7fbc6c09ac40 2024-03-29T11:00:34.105 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:34.105 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:34.105 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient(hunting): dump: 2024-03-29T11:00:34.105 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:34.105 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:34.105 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.106 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.106 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:34.106 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:34.106 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:34.106 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:34.106 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:34.106 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:34.106 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 1 -- 172.21.15.137:0/581374179 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1025620171 0 0) 0x7fbc6c136aa0 con 0x7fbc6c140600 2024-03-29T11:00:34.106 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 1 -- 172.21.15.137:0/581374179 >> v1:172.21.15.71:6790/0 conn(0x7fbc6c1d7bd0 legacy=0x7fbc6c1d8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:34.107 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 1 -- 172.21.15.137:0/581374179 >> v1:172.21.15.71:6789/0 conn(0x7fbc6c09ac40 legacy=0x7fbc6c142b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:34.107 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:00:34.107 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 1 monclient: found mon.b 2024-03-29T11:00:34.107 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:34.107 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:34.107 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 1 -- 172.21.15.137:0/581374179 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc6c0b1a80 con 0x7fbc6c140600 2024-03-29T11:00:34.107 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient: _finish_auth 0 2024-03-29T11:00:34.107 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:00:34.107 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:34.107 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc74582640 5 monclient: authenticate success, global_id 4147 2024-03-29T11:00:34.108 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc74582640 10 monclient: _renew_subs 2024-03-29T11:00:34.108 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc74582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:34.108 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc74582640 1 -- 172.21.15.137:0/581374179 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc6c13ffa0 con 0x7fbc6c140600 2024-03-29T11:00:34.108 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc74582640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:34.108 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc74582640 10 monclient: _renew_subs 2024-03-29T11:00:34.108 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc74582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:34.108 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc74582640 1 -- 172.21.15.137:0/581374179 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc6c1e3f20 con 0x7fbc6c140600 2024-03-29T11:00:34.108 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc74582640 1 librados: init done 2024-03-29T11:00:34.108 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 1 -- 172.21.15.137:0/581374179 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc64004e10 con 0x7fbc6c140600 2024-03-29T11:00:34.108 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:34.108 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 1 -- 172.21.15.137:0/581374179 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fbc64005120 con 0x7fbc6c140600 2024-03-29T11:00:34.109 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:34.109 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:34.109 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc70b1b640 4 set_mon_vals no callback set 2024-03-29T11:00:34.109 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.106+0000 7fbc62ffd640 10 monclient: dump: 2024-03-29T11:00:34.109 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:34.109 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:34.109 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.109 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.109 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:34.110 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:34.110 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:34.110 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:34.110 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:34.110 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:34.110 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.107+0000 7fbc62ffd640 1 -- 172.21.15.137:0/581374179 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fbc64005610 con 0x7fbc6c140600 2024-03-29T11:00:34.110 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.107+0000 7fbc62ffd640 1 -- 172.21.15.137:0/581374179 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 3303829833 0 0) 0x7fbc64006d90 con 0x7fbc6c140600 2024-03-29T11:00:34.110 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.107+0000 7fbc62ffd640 10 client.4147.objecter ms_dispatch 0x7fbc6c000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:00:34.111 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.107+0000 7fbc62ffd640 3 client.4147.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:00:34.111 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.107+0000 7fbc62ffd640 3 client.4147.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:00:34.111 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.107+0000 7fbc62ffd640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-29T11:00:34.111 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.107+0000 7fbc74582640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:34.111 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.107+0000 7fbc74582640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:34.111 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.107+0000 7fbc74582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:34.111 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.107+0000 7fbc74582640 1 -- 172.21.15.137:0/581374179 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc54005080 con 0x7fbc6c140600 2024-03-29T11:00:34.111 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.109+0000 7fbc62ffd640 1 -- 172.21.15.137:0/581374179 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fbc64006850 con 0x7fbc6c140600 2024-03-29T11:00:34.111 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.109+0000 7fbc62ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:34.111 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.109+0000 7fbc62ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:34.160 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:00:34.179+0000 7f5976230040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:00:34.164 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.183+0000 7fbc74582640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "aa3fdfbe-4e06-411f-9b6e-88732ca48f90", "id": 6}] 2024-03-29T11:00:34.164 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.184+0000 7fbc74582640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "aa3fdfbe-4e06-411f-9b6e-88732ca48f90", "id": 6}] 2024-03-29T11:00:34.164 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.184+0000 7fbc74582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:34.164 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.184+0000 7fbc74582640 1 -- 172.21.15.137:0/581374179 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd new", "uuid": "aa3fdfbe-4e06-411f-9b6e-88732ca48f90", "id": 6} v 0) v1 -- 0x7fbc540058b0 con 0x7fbc6c140600 2024-03-29T11:00:34.168 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.188+0000 7fbc62ffd640 1 -- 172.21.15.137:0/581374179 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "aa3fdfbe-4e06-411f-9b6e-88732ca48f90", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 2248501092 0 1113811849) 0x7fbc64006ac0 con 0x7fbc6c140600 2024-03-29T11:00:34.168 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.188+0000 7fbc62ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "aa3fdfbe-4e06-411f-9b6e-88732ca48f90", "id": 6}] 2024-03-29T11:00:34.168 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.188+0000 7fbc62ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:34.168 INFO:teuthology.orchestra.run.smithi137.stdout:6 2024-03-29T11:00:34.169 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.188+0000 7fbc74582640 10 librados: watch_flush enter 2024-03-29T11:00:34.169 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.188+0000 7fbc74582640 10 librados: watch_flush exit 2024-03-29T11:00:34.169 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.188+0000 7fbc74582640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-29T11:00:34.169 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.188+0000 7fbc74582640 10 client.4147.objecter successfully canceled tick 2024-03-29T11:00:34.170 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.189+0000 7fbc74582640 10 monclient: shutdown 2024-03-29T11:00:34.170 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.189+0000 7fbc74582640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:34.170 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.189+0000 7fbc74582640 1 -- 172.21.15.137:0/581374179 >> v1:172.21.15.137:6789/0 conn(0x7fbc6c140600 legacy=0x7fbc6c140a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:34.170 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.189+0000 7fbc74582640 1 -- 172.21.15.137:0/581374179 shutdown_connections 2024-03-29T11:00:34.170 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.189+0000 7fbc74582640 1 -- 172.21.15.137:0/581374179 >> 172.21.15.137:0/581374179 conn(0x7fbc6c066b00 msgr2=0x7fbc6c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:34.170 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.189+0000 7fbc74582640 1 -- 172.21.15.137:0/581374179 shutdown_connections 2024-03-29T11:00:34.170 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.189+0000 7fbc74582640 1 -- 172.21.15.137:0/581374179 wait complete. 2024-03-29T11:00:34.170 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.189+0000 7fbc74582640 1 librados: shutdown 2024-03-29T11:00:34.179 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new a7c11f95-50b1-4b51-aba5-4fe2e78c0f24 7 2024-03-29T11:00:34.186 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T11:00:34.207+0000 7fab49e36040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:00:34.296 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.299+0000 7f2bc0972640 10 monclient: get_monmap_and_config 2024-03-29T11:00:34.297 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.299+0000 7f2bc0972640 10 monclient: build_initial_monmap 2024-03-29T11:00:34.297 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.299+0000 7f2bc0972640 10 monclient: monmap: 2024-03-29T11:00:34.297 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 0 2024-03-29T11:00:34.297 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:34.297 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T11:00:34.300854+0000 2024-03-29T11:00:34.297 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T11:00:34.300854+0000 2024-03-29T11:00:34.297 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:34.297 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:34.297 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:34.298 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:00:34.298 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:00:34.298 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:34.298 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.300+0000 7f2bc0972640 1 Processor -- start 2024-03-29T11:00:34.298 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.300+0000 7f2bc0972640 1 -- start start 2024-03-29T11:00:34.298 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.300+0000 7f2bc0972640 10 monclient: init 2024-03-29T11:00:34.298 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.300+0000 7f2bc0972640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:34.298 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bc0972640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:00:34.298 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bc0972640 10 monclient(hunting): picked mon.noname-c con 0x7f2bbc09ac40 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:34.298 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bc0972640 10 monclient(hunting): picked mon.noname-a con 0x7f2bbc09c870 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:34.298 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bc0972640 10 monclient(hunting): picked mon.noname-b con 0x7f2bbc0b4360 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:34.299 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bc0972640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bbc09dec0 con 0x7f2bbc09c870 2024-03-29T11:00:34.299 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bc0972640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bbc09da30 con 0x7f2bbc09ac40 2024-03-29T11:00:34.299 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bc0972640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bbc0b39c0 con 0x7f2bbc0b4360 2024-03-29T11:00:34.299 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bc0972640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:34.299 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bc0972640 10 monclient(hunting): authenticate will time out at 779.635106s 2024-03-29T11:00:34.299 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bba59c640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f2bbc09ac40 0x7f2bbc0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.137:46960/0 (socket says 172.21.15.137:46960) 2024-03-29T11:00:34.299 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bba59c640 1 -- 172.21.15.137:0/2704691654 learned_addr learned my addr 172.21.15.137:0/2704691654 (peer_addr_for_me v1:172.21.15.137:0/0) 2024-03-29T11:00:34.299 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 1 -- 172.21.15.137:0/2704691654 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2459251033 0 0) 0x7f2bbc09da30 con 0x7f2bbc09ac40 2024-03-29T11:00:34.299 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 10 monclient(hunting): _negotiate 2024-03-29T11:00:34.299 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:34.299 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:34.300 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 10 monclient(hunting): my global_id is 4150 2024-03-29T11:00:34.300 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 1 -- 172.21.15.137:0/2704691654 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ba4003520 con 0x7f2bbc09ac40 2024-03-29T11:00:34.300 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 1 -- 172.21.15.137:0/2704691654 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3380185259 0 0) 0x7f2bb0001110 con 0x7f2bbc09c870 2024-03-29T11:00:34.300 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 10 monclient(hunting): _negotiate 2024-03-29T11:00:34.300 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:34.300 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:34.300 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:00:34.300 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 1 -- 172.21.15.137:0/2704691654 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bbc09da30 con 0x7f2bbc09c870 2024-03-29T11:00:34.300 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 1 -- 172.21.15.137:0/2704691654 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 785505787 0 0) 0x7f2bbc0b39c0 con 0x7f2bbc0b4360 2024-03-29T11:00:34.300 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 10 monclient(hunting): _negotiate 2024-03-29T11:00:34.300 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:34.300 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:34.301 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 10 monclient(hunting): my global_id is 4154 2024-03-29T11:00:34.301 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 1 -- 172.21.15.137:0/2704691654 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bb0001110 con 0x7f2bbc0b4360 2024-03-29T11:00:34.301 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 1 -- 172.21.15.137:0/2704691654 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 260624015 0 0) 0x7f2ba4003520 con 0x7f2bbc09ac40 2024-03-29T11:00:34.301 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 1 -- 172.21.15.137:0/2704691654 >> v1:172.21.15.71:6790/0 conn(0x7f2bbc0b4360 legacy=0x7f2bbc14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:34.301 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 1 -- 172.21.15.137:0/2704691654 >> v1:172.21.15.71:6789/0 conn(0x7f2bbc09c870 legacy=0x7f2bbc13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:34.301 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 10 monclient: _finish_hunting 0 2024-03-29T11:00:34.301 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 1 monclient: found mon.noname-c 2024-03-29T11:00:34.301 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:34.301 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.137:6789/0 2024-03-29T11:00:34.301 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 1 -- 172.21.15.137:0/2704691654 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bbc0b5160 con 0x7f2bbc09ac40 2024-03-29T11:00:34.301 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 10 monclient: _finish_auth 0 2024-03-29T11:00:34.302 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 10 monclient: _check_auth_tickets 2024-03-29T11:00:34.302 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:34.302 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 1 -- 172.21.15.137:0/2704691654 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2bac001690 con 0x7f2bbc09ac40 2024-03-29T11:00:34.302 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bc0972640 5 monclient: authenticate success, global_id 4150 2024-03-29T11:00:34.302 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bc0972640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:34.302 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:34.302 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:34.302 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 10 monclient: dump: 2024-03-29T11:00:34.303 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:34.303 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:34.303 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.303 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.303 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:34.303 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:34.303 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:34.303 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:34.303 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:34.303 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:34.303 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bc0972640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:34.303 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 1 -- 172.21.15.137:0/2704691654 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2bac001870 con 0x7f2bbc09ac40 2024-03-29T11:00:34.304 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:34.304 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bc0972640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:34.304 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 1 -- 172.21.15.137:0/2704691654 <== mon.1 v1:172.21.15.137:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2bac001c80 con 0x7f2bbc09ac40 2024-03-29T11:00:34.304 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:34.304 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bc0972640 4 set_mon_vals no callback set 2024-03-29T11:00:34.304 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:34.304 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.301+0000 7f2bb959a640 10 monclient: dump: 2024-03-29T11:00:34.304 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:34.304 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:34.304 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.304 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.304 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:34.304 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:34.304 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:34.304 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:34.305 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:34.305 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:34.305 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.302+0000 7f2bc0972640 10 monclient: shutdown 2024-03-29T11:00:34.305 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.302+0000 7f2bc0972640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:34.305 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.302+0000 7f2bc0972640 1 -- 172.21.15.137:0/2704691654 >> v1:172.21.15.137:6789/0 conn(0x7f2bbc09ac40 legacy=0x7f2bbc0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:34.305 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.302+0000 7f2bc0972640 1 -- 172.21.15.137:0/2704691654 shutdown_connections 2024-03-29T11:00:34.305 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.302+0000 7f2bc0972640 1 -- 172.21.15.137:0/2704691654 >> 172.21.15.137:0/2704691654 conn(0x7f2bbc066b00 msgr2=0x7f2bbc066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:34.305 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.302+0000 7f2bc0972640 1 -- 172.21.15.137:0/2704691654 shutdown_connections 2024-03-29T11:00:34.305 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.302+0000 7f2bc0972640 1 -- 172.21.15.137:0/2704691654 wait complete. 2024-03-29T11:00:34.305 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.303+0000 7f2bc0972640 10 monclient: build_initial_monmap 2024-03-29T11:00:34.305 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.303+0000 7f2bc0972640 10 monclient: monmap: 2024-03-29T11:00:34.305 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 0 2024-03-29T11:00:34.305 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:34.305 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 0.000000 2024-03-29T11:00:34.305 INFO:teuthology.orchestra.run.smithi137.stderr:created 0.000000 2024-03-29T11:00:34.306 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:34.306 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:34.306 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:34.306 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:00:34.306 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:00:34.306 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:34.306 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.303+0000 7f2bc0972640 1 librados: starting msgr at 2024-03-29T11:00:34.306 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.303+0000 7f2bc0972640 1 librados: starting objecter 2024-03-29T11:00:34.306 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.303+0000 7f2bc0972640 1 Processor -- start 2024-03-29T11:00:34.306 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.303+0000 7f2bc0972640 1 -- start start 2024-03-29T11:00:34.306 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.303+0000 7f2bc0972640 1 librados: setting wanted keys 2024-03-29T11:00:34.306 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.303+0000 7f2bc0972640 1 librados: calling monclient init 2024-03-29T11:00:34.306 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.303+0000 7f2bc0972640 10 monclient: init 2024-03-29T11:00:34.306 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2bc0972640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:34.306 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2bc0972640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:00:34.307 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2bc0972640 10 monclient(hunting): picked mon.noname-b con 0x7f2bbc09ac40 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:34.307 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2bc0972640 10 monclient(hunting): picked mon.noname-a con 0x7f2bbc0b1280 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:34.307 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2bc0972640 10 monclient(hunting): picked mon.noname-c con 0x7f2bbc1d6ed0 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:34.307 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2bc0972640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bbc1422a0 con 0x7f2bbc0b1280 2024-03-29T11:00:34.307 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2bc0972640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bbc142f50 con 0x7f2bbc09ac40 2024-03-29T11:00:34.307 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2bc0972640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bbc1ccab0 con 0x7f2bbc1d6ed0 2024-03-29T11:00:34.307 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2bc0972640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:34.307 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2bc0972640 10 monclient(hunting): authenticate will time out at 779.638189s 2024-03-29T11:00:34.307 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2bba59c640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f2bbc09ac40 0x7f2bbc141ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.137:46966/0 (socket says 172.21.15.137:46966) 2024-03-29T11:00:34.307 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2bba59c640 1 -- 172.21.15.137:0/3916924256 learned_addr learned my addr 172.21.15.137:0/3916924256 (peer_addr_for_me v1:172.21.15.137:0/0) 2024-03-29T11:00:34.307 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 client.?.objecter ms_handle_connect 0x7f2bbc09ac40 2024-03-29T11:00:34.307 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:34.307 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 1 -- 172.21.15.137:0/3916924256 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3080573084 0 0) 0x7f2bbc142f50 con 0x7f2bbc09ac40 2024-03-29T11:00:34.307 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:34.307 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:34.308 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:34.308 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 monclient(hunting): my global_id is 4153 2024-03-29T11:00:34.308 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 1 -- 172.21.15.137:0/3916924256 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b9c004eb0 con 0x7f2bbc09ac40 2024-03-29T11:00:34.308 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 client.?.objecter ms_handle_connect 0x7f2bbc0b1280 2024-03-29T11:00:34.308 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:34.308 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 client.?.objecter ms_handle_connect 0x7f2bbc1d6ed0 2024-03-29T11:00:34.308 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:34.308 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 1 -- 172.21.15.137:0/3916924256 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3079583185 0 0) 0x7f2bbc1422a0 con 0x7f2bbc0b1280 2024-03-29T11:00:34.308 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:34.308 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:34.308 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:34.308 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 monclient(hunting): my global_id is 4158 2024-03-29T11:00:34.308 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 1 -- 172.21.15.137:0/3916924256 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bbc142f50 con 0x7f2bbc0b1280 2024-03-29T11:00:34.308 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 1 -- 172.21.15.137:0/3916924256 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 971409926 0 0) 0x7f2bbc1ccab0 con 0x7f2bbc1d6ed0 2024-03-29T11:00:34.308 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:34.309 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:34.309 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:34.309 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 monclient(hunting): my global_id is 4157 2024-03-29T11:00:34.309 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 1 -- 172.21.15.137:0/3916924256 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bbc1422a0 con 0x7f2bbc1d6ed0 2024-03-29T11:00:34.309 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 1 -- 172.21.15.137:0/3916924256 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1576218645 0 0) 0x7f2b9c004eb0 con 0x7f2bbc09ac40 2024-03-29T11:00:34.309 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 1 -- 172.21.15.137:0/3916924256 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2bbc1ccab0 con 0x7f2bbc09ac40 2024-03-29T11:00:34.309 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 1 -- 172.21.15.137:0/3916924256 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2bac001bc0 con 0x7f2bbc09ac40 2024-03-29T11:00:34.309 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:34.309 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:34.309 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 monclient(hunting): dump: 2024-03-29T11:00:34.309 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:34.309 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:34.309 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.309 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.310 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:34.310 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:34.310 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:34.310 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:34.310 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:34.310 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:34.310 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:34.310 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 1 -- 172.21.15.137:0/3916924256 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1734704414 0 0) 0x7f2bbc142f50 con 0x7f2bbc0b1280 2024-03-29T11:00:34.310 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 1 -- 172.21.15.137:0/3916924256 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b9c004eb0 con 0x7f2bbc0b1280 2024-03-29T11:00:34.310 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 1 -- 172.21.15.137:0/3916924256 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2bb0001920 con 0x7f2bbc0b1280 2024-03-29T11:00:34.310 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:34.310 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:34.310 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 monclient(hunting): dump: 2024-03-29T11:00:34.310 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:34.310 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:34.311 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.311 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.311 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:34.311 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:34.311 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:34.311 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:34.311 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:34.311 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:34.311 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 1 -- 172.21.15.137:0/3916924256 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1387304667 0 0) 0x7f2bbc1422a0 con 0x7f2bbc1d6ed0 2024-03-29T11:00:34.311 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 1 -- 172.21.15.137:0/3916924256 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2bbc142f50 con 0x7f2bbc1d6ed0 2024-03-29T11:00:34.311 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 1 -- 172.21.15.137:0/3916924256 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2bb4001a80 con 0x7f2bbc1d6ed0 2024-03-29T11:00:34.311 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:34.311 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:34.311 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 monclient(hunting): dump: 2024-03-29T11:00:34.311 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:34.312 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:34.312 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.312 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.312 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:34.312 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:34.312 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:34.312 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:34.312 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:34.312 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:34.312 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 1 -- 172.21.15.137:0/3916924256 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 97131132 0 0) 0x7f2bbc1ccab0 con 0x7f2bbc09ac40 2024-03-29T11:00:34.312 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 1 -- 172.21.15.137:0/3916924256 >> v1:172.21.15.71:6790/0 conn(0x7f2bbc1d6ed0 legacy=0x7f2bbc13adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:34.312 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 1 -- 172.21.15.137:0/3916924256 >> v1:172.21.15.71:6789/0 conn(0x7f2bbc0b1280 legacy=0x7f2bbc142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:34.312 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:00:34.312 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 1 monclient: found mon.b 2024-03-29T11:00:34.312 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:34.313 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:34.313 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 1 -- 172.21.15.137:0/3916924256 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bbc1ccc40 con 0x7f2bbc09ac40 2024-03-29T11:00:34.313 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 monclient: _finish_auth 0 2024-03-29T11:00:34.313 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:00:34.313 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2baaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:34.313 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2bc0972640 5 monclient: authenticate success, global_id 4153 2024-03-29T11:00:34.313 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2bc0972640 10 monclient: _renew_subs 2024-03-29T11:00:34.313 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2bc0972640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:34.313 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.304+0000 7f2bc0972640 1 -- 172.21.15.137:0/3916924256 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bbc13b4d0 con 0x7f2bbc09ac40 2024-03-29T11:00:34.313 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.305+0000 7f2bc0972640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:34.313 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.305+0000 7f2bc0972640 10 monclient: _renew_subs 2024-03-29T11:00:34.313 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.305+0000 7f2bc0972640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:34.313 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.305+0000 7f2bc0972640 1 -- 172.21.15.137:0/3916924256 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bbc1e3720 con 0x7f2bbc09ac40 2024-03-29T11:00:34.314 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.305+0000 7f2bc0972640 1 librados: init done 2024-03-29T11:00:34.314 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.305+0000 7f2baaffd640 1 -- 172.21.15.137:0/3916924256 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2bac004bf0 con 0x7f2bbc09ac40 2024-03-29T11:00:34.314 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.305+0000 7f2baaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:34.314 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.305+0000 7f2baaffd640 1 -- 172.21.15.137:0/3916924256 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2bac004f00 con 0x7f2bbc09ac40 2024-03-29T11:00:34.314 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.305+0000 7f2baaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:34.314 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.305+0000 7f2baaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:34.314 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.305+0000 7f2bb8d99640 4 set_mon_vals no callback set 2024-03-29T11:00:34.314 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.305+0000 7f2baaffd640 10 monclient: dump: 2024-03-29T11:00:34.314 INFO:teuthology.orchestra.run.smithi137.stderr:epoch 1 2024-03-29T11:00:34.314 INFO:teuthology.orchestra.run.smithi137.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:34.314 INFO:teuthology.orchestra.run.smithi137.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.314 INFO:teuthology.orchestra.run.smithi137.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.314 INFO:teuthology.orchestra.run.smithi137.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:34.314 INFO:teuthology.orchestra.run.smithi137.stderr:election_strategy: 1 2024-03-29T11:00:34.314 INFO:teuthology.orchestra.run.smithi137.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:34.315 INFO:teuthology.orchestra.run.smithi137.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:34.315 INFO:teuthology.orchestra.run.smithi137.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:34.315 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:00:34.315 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.305+0000 7f2baaffd640 1 -- 172.21.15.137:0/3916924256 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2bac0053f0 con 0x7f2bbc09ac40 2024-03-29T11:00:34.315 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.305+0000 7f2baaffd640 1 -- 172.21.15.137:0/3916924256 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 1889072320 0 0) 0x7f2bac006b90 con 0x7f2bbc09ac40 2024-03-29T11:00:34.315 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.305+0000 7f2baaffd640 10 client.4153.objecter ms_dispatch 0x7f2bbc000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:00:34.315 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.305+0000 7f2baaffd640 3 client.4153.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:00:34.315 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.305+0000 7f2baaffd640 3 client.4153.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:00:34.315 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.305+0000 7f2baaffd640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-29T11:00:34.315 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.305+0000 7f2bc0972640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:34.315 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.305+0000 7f2bc0972640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:34.315 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.305+0000 7f2bc0972640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:34.315 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.305+0000 7f2bc0972640 1 -- 172.21.15.137:0/3916924256 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ba0005080 con 0x7f2bbc09ac40 2024-03-29T11:00:34.315 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.307+0000 7f2baaffd640 1 -- 172.21.15.137:0/3916924256 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2bac0065e0 con 0x7f2bbc09ac40 2024-03-29T11:00:34.316 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.307+0000 7f2baaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:34.316 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.307+0000 7f2baaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:34.316 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:00:34.316+0000 7f5976230040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:00:34.328 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T11:00:34.348+0000 7fab49e36040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:00:34.360 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.380+0000 7f2bc0972640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a7c11f95-50b1-4b51-aba5-4fe2e78c0f24", "id": 7}] 2024-03-29T11:00:34.361 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.380+0000 7f2bc0972640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a7c11f95-50b1-4b51-aba5-4fe2e78c0f24", "id": 7}] 2024-03-29T11:00:34.361 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.380+0000 7f2bc0972640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:34.361 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.380+0000 7f2bc0972640 1 -- 172.21.15.137:0/3916924256 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a7c11f95-50b1-4b51-aba5-4fe2e78c0f24", "id": 7} v 0) v1 -- 0x7f2ba00058b0 con 0x7f2bbc09ac40 2024-03-29T11:00:34.363 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.383+0000 7f2baaffd640 1 -- 172.21.15.137:0/3916924256 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a7c11f95-50b1-4b51-aba5-4fe2e78c0f24", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 1814976636 0 1371665406) 0x7f2bac006850 con 0x7f2bbc09ac40 2024-03-29T11:00:34.363 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.383+0000 7f2baaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a7c11f95-50b1-4b51-aba5-4fe2e78c0f24", "id": 7}] 2024-03-29T11:00:34.363 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.383+0000 7f2baaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:34.363 INFO:teuthology.orchestra.run.smithi137.stdout:7 2024-03-29T11:00:34.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.384+0000 7f2bc0972640 10 librados: watch_flush enter 2024-03-29T11:00:34.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.384+0000 7f2bc0972640 10 librados: watch_flush exit 2024-03-29T11:00:34.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.384+0000 7f2bc0972640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-29T11:00:34.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.384+0000 7f2bc0972640 10 client.4153.objecter successfully canceled tick 2024-03-29T11:00:34.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.384+0000 7f2bc0972640 10 monclient: shutdown 2024-03-29T11:00:34.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.384+0000 7f2bc0972640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:34.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.384+0000 7f2bc0972640 1 -- 172.21.15.137:0/3916924256 >> v1:172.21.15.137:6789/0 conn(0x7f2bbc09ac40 legacy=0x7f2bbc141ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:34.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.384+0000 7f2bc0972640 1 -- 172.21.15.137:0/3916924256 shutdown_connections 2024-03-29T11:00:34.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.384+0000 7f2bc0972640 1 -- 172.21.15.137:0/3916924256 >> 172.21.15.137:0/3916924256 conn(0x7f2bbc066b00 msgr2=0x7f2bbc066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:34.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.384+0000 7f2bc0972640 1 -- 172.21.15.137:0/3916924256 shutdown_connections 2024-03-29T11:00:34.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.384+0000 7f2bc0972640 1 -- 172.21.15.137:0/3916924256 wait complete. 2024-03-29T11:00:34.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-29T11:00:34.384+0000 7f2bc0972640 1 librados: shutdown 2024-03-29T11:00:34.375 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:00:34.375 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:00:34.377 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:00:34.377 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:00:34.377 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:00:34.379 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:00:34.379 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:00:34.380 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:00:34.381 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:00:34.382 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:00:34.382 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:00:34.384 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:00:34.384 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:00:34.384 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:00:34.413 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:00:34.432+0000 7f5976230040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:00:34.417 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:00:34.417 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:00:34.417 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:00:34.419 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:00:34.419 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:00:34.419 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:00:34.421 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:00:34.422 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:00:34.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:00:34.424 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:00:34.424 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:34.452 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T11:00:34.472+0000 7fab49e36040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:00:34.485 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:00:34.504+0000 7f5976230040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:00:34.518 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T11:00:34.538+0000 7fab49e36040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:00:34.534 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.554+0000 7efc318a8640 10 monclient: get_monmap_and_config 2024-03-29T11:00:34.534 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.554+0000 7efc318a8640 10 monclient: build_initial_monmap 2024-03-29T11:00:34.535 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.554+0000 7efc318a8640 10 monclient: monmap: 2024-03-29T11:00:34.535 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-03-29T11:00:34.535 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:34.535 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-29T11:00:34.555875+0000 2024-03-29T11:00:34.535 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-29T11:00:34.555875+0000 2024-03-29T11:00:34.535 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:34.535 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:34.535 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:34.535 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:00:34.535 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:00:34.536 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:00:34.549 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.554+0000 7efc318a8640 1 Processor -- start 2024-03-29T11:00:34.549 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.554+0000 7efc318a8640 1 -- start start 2024-03-29T11:00:34.550 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.554+0000 7efc318a8640 10 monclient: init 2024-03-29T11:00:34.550 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.555+0000 7efc318a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:34.550 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.555+0000 7efc318a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:00:34.550 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.555+0000 7efc318a8640 10 monclient(hunting): picked mon.noname-c con 0x7efc2c09ac40 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:34.550 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.555+0000 7efc318a8640 10 monclient(hunting): picked mon.noname-b con 0x7efc2c09c870 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:34.550 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.555+0000 7efc318a8640 10 monclient(hunting): picked mon.noname-a con 0x7efc2c0b4360 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:34.550 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.555+0000 7efc318a8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc2c09dec0 con 0x7efc2c0b4360 2024-03-29T11:00:34.550 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.555+0000 7efc318a8640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc2c09da30 con 0x7efc2c09ac40 2024-03-29T11:00:34.550 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.555+0000 7efc2b7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7efc2c0b4360 0x7efc2c14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:60396/0 (socket says 172.21.15.71:60396) 2024-03-29T11:00:34.551 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.555+0000 7efc2b7fe640 1 -- 172.21.15.71:0/621908096 learned_addr learned my addr 172.21.15.71:0/621908096 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:00:34.551 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.555+0000 7efc318a8640 1 -- 172.21.15.71:0/621908096 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc2c0b39c0 con 0x7efc2c09c870 2024-03-29T11:00:34.551 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.555+0000 7efc318a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:34.551 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.555+0000 7efc318a8640 10 monclient(hunting): authenticate will time out at 790.814837s 2024-03-29T11:00:34.551 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.555+0000 7efc29ffb640 1 -- 172.21.15.71:0/621908096 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2165092889 0 0) 0x7efc1c001490 con 0x7efc2c0b4360 2024-03-29T11:00:34.551 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.555+0000 7efc29ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:34.551 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.555+0000 7efc29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:34.551 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.555+0000 7efc29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:34.552 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.555+0000 7efc29ffb640 10 monclient(hunting): my global_id is 4185 2024-03-29T11:00:34.552 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.555+0000 7efc29ffb640 1 -- 172.21.15.71:0/621908096 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc14003520 con 0x7efc2c0b4360 2024-03-29T11:00:34.552 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.555+0000 7efc29ffb640 1 -- 172.21.15.71:0/621908096 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989215130 0 0) 0x7efc2c0b39c0 con 0x7efc2c09c870 2024-03-29T11:00:34.552 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.555+0000 7efc29ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:34.552 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.555+0000 7efc29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:34.552 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.555+0000 7efc29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:34.552 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.555+0000 7efc29ffb640 10 monclient(hunting): my global_id is 4184 2024-03-29T11:00:34.552 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.555+0000 7efc29ffb640 1 -- 172.21.15.71:0/621908096 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc1c001490 con 0x7efc2c09c870 2024-03-29T11:00:34.552 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc29ffb640 1 -- 172.21.15.71:0/621908096 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2405548205 0 0) 0x7efc2c09da30 con 0x7efc2c09ac40 2024-03-29T11:00:34.553 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc29ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:34.553 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:34.553 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:34.553 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc29ffb640 10 monclient(hunting): my global_id is 4180 2024-03-29T11:00:34.553 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc29ffb640 1 -- 172.21.15.71:0/621908096 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc2c0b39c0 con 0x7efc2c09ac40 2024-03-29T11:00:34.553 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc29ffb640 1 -- 172.21.15.71:0/621908096 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1570009936 0 0) 0x7efc1c001490 con 0x7efc2c09c870 2024-03-29T11:00:34.553 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc29ffb640 1 -- 172.21.15.71:0/621908096 >> v1:172.21.15.137:6789/0 conn(0x7efc2c09ac40 legacy=0x7efc2c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:34.553 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc29ffb640 1 -- 172.21.15.71:0/621908096 >> v1:172.21.15.71:6789/0 conn(0x7efc2c0b4360 legacy=0x7efc2c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:34.553 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc29ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:00:34.554 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc29ffb640 1 monclient: found mon.noname-b 2024-03-29T11:00:34.554 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc29ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:34.554 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc29ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:00:34.554 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc29ffb640 1 -- 172.21.15.71:0/621908096 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc2c0b5160 con 0x7efc2c09c870 2024-03-29T11:00:34.554 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc29ffb640 10 monclient: _finish_auth 0 2024-03-29T11:00:34.554 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc29ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:00:34.554 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc29ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:34.555 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc29ffb640 1 -- 172.21.15.71:0/621908096 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7efc20001670 con 0x7efc2c09c870 2024-03-29T11:00:34.555 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:34.555 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc29ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:34.555 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc29ffb640 10 monclient: dump: 2024-03-29T11:00:34.555 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-29T11:00:34.555 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:34.555 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.555 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.555 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:34.555 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:34.555 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:34.556 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:34.556 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:34.556 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:00:34.556 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc318a8640 5 monclient: authenticate success, global_id 4184 2024-03-29T11:00:34.556 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc318a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:34.556 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc29ffb640 1 -- 172.21.15.71:0/621908096 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc20001850 con 0x7efc2c09c870 2024-03-29T11:00:34.556 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc29ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:34.556 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc29ffb640 1 -- 172.21.15.71:0/621908096 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7efc20001c60 con 0x7efc2c09c870 2024-03-29T11:00:34.556 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:34.556 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc29ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:34.556 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc29ffb640 10 monclient: dump: 2024-03-29T11:00:34.556 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-29T11:00:34.557 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:34.557 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.557 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.557 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:34.557 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:34.557 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:34.557 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:34.557 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:34.557 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:00:34.557 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc318a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:34.557 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc318a8640 4 set_mon_vals no callback set 2024-03-29T11:00:34.557 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc318a8640 10 monclient: shutdown 2024-03-29T11:00:34.558 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc318a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:34.558 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.556+0000 7efc318a8640 1 -- 172.21.15.71:0/621908096 >> v1:172.21.15.71:6790/0 conn(0x7efc2c09c870 legacy=0x7efc2c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:34.558 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.557+0000 7efc318a8640 1 -- 172.21.15.71:0/621908096 shutdown_connections 2024-03-29T11:00:34.558 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.557+0000 7efc318a8640 1 -- 172.21.15.71:0/621908096 >> 172.21.15.71:0/621908096 conn(0x7efc2c066b00 msgr2=0x7efc2c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:34.558 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.557+0000 7efc318a8640 1 -- 172.21.15.71:0/621908096 shutdown_connections 2024-03-29T11:00:34.558 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.557+0000 7efc318a8640 1 -- 172.21.15.71:0/621908096 wait complete. 2024-03-29T11:00:34.558 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.557+0000 7efc318a8640 10 monclient: build_initial_monmap 2024-03-29T11:00:34.558 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.557+0000 7efc318a8640 10 monclient: monmap: 2024-03-29T11:00:34.558 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-03-29T11:00:34.558 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:34.558 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 0.000000 2024-03-29T11:00:34.558 INFO:teuthology.orchestra.run.smithi071.stderr:created 0.000000 2024-03-29T11:00:34.559 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:34.559 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:34.559 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:34.559 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:00:34.559 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:00:34.559 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:00:34.559 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.558+0000 7efc318a8640 1 librados: starting msgr at 2024-03-29T11:00:34.559 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.558+0000 7efc318a8640 1 librados: starting objecter 2024-03-29T11:00:34.559 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.558+0000 7efc318a8640 1 Processor -- start 2024-03-29T11:00:34.559 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.558+0000 7efc318a8640 1 -- start start 2024-03-29T11:00:34.559 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.558+0000 7efc318a8640 1 librados: setting wanted keys 2024-03-29T11:00:34.559 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.558+0000 7efc318a8640 1 librados: calling monclient init 2024-03-29T11:00:34.560 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.558+0000 7efc318a8640 10 monclient: init 2024-03-29T11:00:34.560 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc318a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:34.560 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc318a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:00:34.560 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc318a8640 10 monclient(hunting): picked mon.noname-b con 0x7efc2c09ac40 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:34.560 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc318a8640 10 monclient(hunting): picked mon.noname-a con 0x7efc2c1419a0 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:34.560 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc318a8640 10 monclient(hunting): picked mon.noname-c con 0x7efc2c1396c0 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:34.560 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc318a8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc2c1da610 con 0x7efc2c1419a0 2024-03-29T11:00:34.560 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc318a8640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc2c1da230 con 0x7efc2c09ac40 2024-03-29T11:00:34.560 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc2a7fc640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7efc2c1419a0 0x7efc2c138fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:60412/0 (socket says 172.21.15.71:60412) 2024-03-29T11:00:34.560 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc2a7fc640 1 -- 172.21.15.71:0/1450206757 learned_addr learned my addr 172.21.15.71:0/1450206757 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:00:34.560 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc2b7fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7efc2c1396c0 0x7efc2c1e25d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:38322/0 (socket says 172.21.15.71:38322) 2024-03-29T11:00:34.560 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc318a8640 1 -- 172.21.15.71:0/1450206757 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc2c1da3c0 con 0x7efc2c1396c0 2024-03-29T11:00:34.561 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc318a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:34.561 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc318a8640 10 monclient(hunting): authenticate will time out at 790.818383s 2024-03-29T11:00:34.561 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 client.?.objecter ms_handle_connect 0x7efc2c1396c0 2024-03-29T11:00:34.561 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:34.561 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 client.?.objecter ms_handle_connect 0x7efc2c1419a0 2024-03-29T11:00:34.561 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:34.561 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 client.?.objecter ms_handle_connect 0x7efc2c09ac40 2024-03-29T11:00:34.561 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:34.561 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 1 -- 172.21.15.71:0/1450206757 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4138768719 0 0) 0x7efc2c1da610 con 0x7efc2c1419a0 2024-03-29T11:00:34.561 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:34.561 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:34.561 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:34.562 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:00:34.562 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 1 -- 172.21.15.71:0/1450206757 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc18004eb0 con 0x7efc2c1419a0 2024-03-29T11:00:34.562 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 1 -- 172.21.15.71:0/1450206757 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707156848 0 0) 0x7efc2c1da3c0 con 0x7efc2c1396c0 2024-03-29T11:00:34.562 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:34.562 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:34.562 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:34.562 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 monclient(hunting): my global_id is 4187 2024-03-29T11:00:34.562 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 1 -- 172.21.15.71:0/1450206757 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc18006230 con 0x7efc2c1396c0 2024-03-29T11:00:34.562 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 1 -- 172.21.15.71:0/1450206757 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911269893 0 0) 0x7efc2c1da230 con 0x7efc2c09ac40 2024-03-29T11:00:34.562 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:34.562 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:34.562 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:34.562 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 monclient(hunting): my global_id is 4183 2024-03-29T11:00:34.563 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 1 -- 172.21.15.71:0/1450206757 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc2c1da3c0 con 0x7efc2c09ac40 2024-03-29T11:00:34.563 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 1 -- 172.21.15.71:0/1450206757 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3580014839 0 0) 0x7efc18006230 con 0x7efc2c1396c0 2024-03-29T11:00:34.563 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 1 -- 172.21.15.71:0/1450206757 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc2c1da230 con 0x7efc2c1396c0 2024-03-29T11:00:34.563 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 1 -- 172.21.15.71:0/1450206757 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7efc1c001a90 con 0x7efc2c1396c0 2024-03-29T11:00:34.563 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:34.563 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:34.563 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 monclient(hunting): dump: 2024-03-29T11:00:34.563 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-29T11:00:34.563 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:34.563 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.563 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.563 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:34.564 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:34.564 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:34.564 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:34.564 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:34.564 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:00:34.564 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:34.564 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 1 -- 172.21.15.71:0/1450206757 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2096695207 0 0) 0x7efc2c1da3c0 con 0x7efc2c09ac40 2024-03-29T11:00:34.564 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 1 -- 172.21.15.71:0/1450206757 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc18006230 con 0x7efc2c09ac40 2024-03-29T11:00:34.564 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 1 -- 172.21.15.71:0/1450206757 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2427582847 0 0) 0x7efc18004eb0 con 0x7efc2c1419a0 2024-03-29T11:00:34.564 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 1 -- 172.21.15.71:0/1450206757 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc2c1da3c0 con 0x7efc2c1419a0 2024-03-29T11:00:34.564 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 1 -- 172.21.15.71:0/1450206757 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7efc24001830 con 0x7efc2c09ac40 2024-03-29T11:00:34.564 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:34.565 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:34.565 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 monclient(hunting): dump: 2024-03-29T11:00:34.565 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-29T11:00:34.565 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:34.565 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.565 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.565 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:34.565 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:34.565 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:34.565 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:34.565 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:34.565 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:00:34.566 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 1 -- 172.21.15.71:0/1450206757 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7efc20001ba0 con 0x7efc2c1419a0 2024-03-29T11:00:34.566 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:34.566 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:34.566 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 monclient(hunting): dump: 2024-03-29T11:00:34.566 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-29T11:00:34.566 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:34.566 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.566 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.566 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:34.566 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:34.566 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:34.567 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:34.567 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:34.567 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:00:34.567 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 1 -- 172.21.15.71:0/1450206757 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3217508734 0 0) 0x7efc2c1da230 con 0x7efc2c1396c0 2024-03-29T11:00:34.567 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 1 -- 172.21.15.71:0/1450206757 >> v1:172.21.15.137:6789/0 conn(0x7efc2c09ac40 legacy=0x7efc2c1dab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:34.567 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 1 -- 172.21.15.71:0/1450206757 >> v1:172.21.15.71:6789/0 conn(0x7efc2c1419a0 legacy=0x7efc2c138fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:34.567 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 monclient: _finish_hunting 0 2024-03-29T11:00:34.567 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 1 monclient: found mon.c 2024-03-29T11:00:34.567 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:34.567 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:00:34.567 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 1 -- 172.21.15.71:0/1450206757 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc2c140590 con 0x7efc2c1396c0 2024-03-29T11:00:34.567 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 monclient: _finish_auth 0 2024-03-29T11:00:34.567 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 10 monclient: _check_auth_tickets 2024-03-29T11:00:34.568 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:34.568 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc318a8640 5 monclient: authenticate success, global_id 4187 2024-03-29T11:00:34.568 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.559+0000 7efc318a8640 10 monclient: _renew_subs 2024-03-29T11:00:34.568 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.560+0000 7efc318a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:00:34.568 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.560+0000 7efc13fff640 1 -- 172.21.15.71:0/1450206757 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc1c001f80 con 0x7efc2c1396c0 2024-03-29T11:00:34.568 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.560+0000 7efc318a8640 1 -- 172.21.15.71:0/1450206757 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc2c140780 con 0x7efc2c1396c0 2024-03-29T11:00:34.568 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.560+0000 7efc13fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:34.568 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.560+0000 7efc13fff640 1 -- 172.21.15.71:0/1450206757 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7efc1c004d00 con 0x7efc2c1396c0 2024-03-29T11:00:34.568 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.560+0000 7efc13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:34.568 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.560+0000 7efc13fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:34.568 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.560+0000 7efc13fff640 10 monclient: dump: 2024-03-29T11:00:34.569 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-29T11:00:34.569 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:34.569 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.569 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:34.569 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:34.569 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:34.569 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:34.569 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:34.569 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:34.569 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:00:34.569 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.560+0000 7efc297fa640 4 set_mon_vals no callback set 2024-03-29T11:00:34.569 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.560+0000 7efc318a8640 10 client.4187.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:34.569 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.560+0000 7efc318a8640 10 monclient: _renew_subs 2024-03-29T11:00:34.570 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.560+0000 7efc318a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:00:34.570 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.560+0000 7efc318a8640 1 -- 172.21.15.71:0/1450206757 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc2c1415b0 con 0x7efc2c1396c0 2024-03-29T11:00:34.570 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.560+0000 7efc318a8640 1 librados: init done 2024-03-29T11:00:34.570 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.560+0000 7efc13fff640 1 -- 172.21.15.71:0/1450206757 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7efc1c0051f0 con 0x7efc2c1396c0 2024-03-29T11:00:34.570 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.560+0000 7efc13fff640 1 -- 172.21.15.71:0/1450206757 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7efc1c005b70 con 0x7efc2c1396c0 2024-03-29T11:00:34.570 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.560+0000 7efc13fff640 10 client.4187.objecter ms_dispatch 0x7efc2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:34.570 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.560+0000 7efc13fff640 3 client.4187.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:34.570 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.560+0000 7efc13fff640 3 client.4187.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:34.570 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.560+0000 7efc13fff640 20 client.4187.objecter dump_active .. 0 homeless 2024-03-29T11:00:34.570 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.561+0000 7efc318a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:34.570 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.561+0000 7efc318a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:34.570 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.561+0000 7efc318a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:00:34.571 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.561+0000 7efc318a8640 1 -- 172.21.15.71:0/1450206757 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc2c09b020 con 0x7efc2c1396c0 2024-03-29T11:00:34.571 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.563+0000 7efc13fff640 1 -- 172.21.15.71:0/1450206757 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7efc1c0054a0 con 0x7efc2c1396c0 2024-03-29T11:00:34.571 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.563+0000 7efc13fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:34.571 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.563+0000 7efc13fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:34.605 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:00:34.624+0000 7f5976230040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:00:34.606 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.624+0000 7efc318a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:34.606 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.624+0000 7efc318a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:34.606 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.624+0000 7efc318a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:00:34.606 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.624+0000 7efc318a8640 1 -- 172.21.15.71:0/1450206757 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc2c04f730 con 0x7efc2c1396c0 2024-03-29T11:00:34.606 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.625+0000 7efc13fff640 1 -- 172.21.15.71:0/1450206757 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7efc1c005740 con 0x7efc2c1396c0 2024-03-29T11:00:34.606 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.625+0000 7efc13fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:34.606 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.625+0000 7efc13fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:34.607 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:00:34.607 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:34.608 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.625+0000 7efc318a8640 10 librados: watch_flush enter 2024-03-29T11:00:34.608 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.625+0000 7efc318a8640 10 librados: watch_flush exit 2024-03-29T11:00:34.608 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.626+0000 7efc318a8640 20 client.4187.objecter shutdown clearing up homeless session... 2024-03-29T11:00:34.609 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.626+0000 7efc318a8640 10 client.4187.objecter successfully canceled tick 2024-03-29T11:00:34.609 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.626+0000 7efc318a8640 10 monclient: shutdown 2024-03-29T11:00:34.609 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.626+0000 7efc318a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:34.609 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.626+0000 7efc318a8640 1 -- 172.21.15.71:0/1450206757 >> v1:172.21.15.71:6790/0 conn(0x7efc2c1396c0 legacy=0x7efc2c1e25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:34.609 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.626+0000 7efc318a8640 1 -- 172.21.15.71:0/1450206757 shutdown_connections 2024-03-29T11:00:34.609 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.626+0000 7efc318a8640 1 -- 172.21.15.71:0/1450206757 >> 172.21.15.71:0/1450206757 conn(0x7efc2c066b00 msgr2=0x7efc2c13a230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:34.609 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.626+0000 7efc318a8640 1 -- 172.21.15.71:0/1450206757 shutdown_connections 2024-03-29T11:00:34.609 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.626+0000 7efc318a8640 1 -- 172.21.15.71:0/1450206757 wait complete. 2024-03-29T11:00:34.609 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:00:34.626+0000 7efc318a8640 1 librados: shutdown 2024-03-29T11:00:34.617 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:00:34.617 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:00:34.631 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T11:00:34.650+0000 7fab49e36040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:00:34.702 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:00:34.721+0000 7f5976230040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:00:34.723 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T11:00:34.743+0000 7fab49e36040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:00:34.859 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:00:34.878+0000 7f5976230040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:00:34.876 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T11:00:34.896+0000 7fab49e36040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:00:34.901 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:00:34.920+0000 7f5976230040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:00:34.918 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T11:00:34.938+0000 7fab49e36040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:00:34.987 INFO:tasks.ceph.osd.1.smithi071.stderr:2024-03-29T11:00:35.006+0000 7f09baa0d640 -1 Falling back to public interface 2024-03-29T11:00:34.988 INFO:tasks.ceph.osd.0.smithi071.stderr:2024-03-29T11:00:35.007+0000 7f35e0615640 -1 Falling back to public interface 2024-03-29T11:00:34.991 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-03-29T11:00:35.011+0000 7f5bda345640 -1 Falling back to public interface 2024-03-29T11:00:35.003 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-03-29T11:00:35.022+0000 7f7e00611640 -1 Falling back to public interface 2024-03-29T11:00:35.019 INFO:tasks.ceph.osd.6.smithi137.stderr:2024-03-29T11:00:35.039+0000 7f322cc17640 -1 Falling back to public interface 2024-03-29T11:00:35.020 INFO:tasks.ceph.osd.5.smithi137.stderr:2024-03-29T11:00:35.040+0000 7f062b023640 -1 Falling back to public interface 2024-03-29T11:00:35.027 INFO:tasks.ceph.osd.7.smithi137.stderr:2024-03-29T11:00:35.047+0000 7f7fe9149640 -1 Falling back to public interface 2024-03-29T11:00:35.032 INFO:tasks.ceph.osd.4.smithi137.stderr:2024-03-29T11:00:35.053+0000 7f22cba1c640 -1 Falling back to public interface 2024-03-29T11:00:35.059 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:00:35.078+0000 7f5976230040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:00:35.075 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T11:00:35.095+0000 7fab49e36040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:00:35.423 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:35.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.532+0000 7f0016214640 10 monclient: get_monmap_and_config 2024-03-29T11:00:35.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.532+0000 7f0016214640 10 monclient: build_initial_monmap 2024-03-29T11:00:35.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.532+0000 7f0016214640 10 monclient: monmap: 2024-03-29T11:00:35.514 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:00:35.514 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:35.514 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:00:35.534701+0000 2024-03-29T11:00:35.514 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:00:35.534701+0000 2024-03-29T11:00:35.514 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:35.514 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:35.514 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:35.514 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:00:35.514 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:00:35.514 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:00:35.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.533+0000 7f0016214640 1 Processor -- start 2024-03-29T11:00:35.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.533+0000 7f0016214640 1 -- start start 2024-03-29T11:00:35.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.533+0000 7f0016214640 10 monclient: init 2024-03-29T11:00:35.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f0016214640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:35.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f0016214640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:00:35.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f0016214640 10 monclient(hunting): picked mon.noname-b con 0x7f00100b4710 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:35.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f0016214640 10 monclient(hunting): picked mon.noname-a con 0x7f00100b3970 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:35.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f0016214640 10 monclient(hunting): picked mon.noname-c con 0x7f001013f390 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:35.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f0016214640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00100af020 con 0x7f00100b3970 2024-03-29T11:00:35.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f0016214640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00100b0910 con 0x7f001013f390 2024-03-29T11:00:35.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f0016214640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00100b0bf0 con 0x7f00100b4710 2024-03-29T11:00:35.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f0016214640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:35.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f0016214640 10 monclient(hunting): authenticate will time out at 791.793452s 2024-03-29T11:00:35.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f0006ffd640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f00100b3970 0x7f00100af4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:60416/0 (socket says 172.21.15.71:60416) 2024-03-29T11:00:35.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f0006ffd640 1 -- 172.21.15.71:0/2450517603 learned_addr learned my addr 172.21.15.71:0/2450517603 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:00:35.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 1 -- 172.21.15.71:0/2450517603 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3070776031 0 0) 0x7f00100b0bf0 con 0x7f00100b4710 2024-03-29T11:00:35.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:35.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:35.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:35.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 10 monclient(hunting): my global_id is 4190 2024-03-29T11:00:35.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 1 -- 172.21.15.71:0/2450517603 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efffc003520 con 0x7f00100b4710 2024-03-29T11:00:35.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 1 -- 172.21.15.71:0/2450517603 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4103555488 0 0) 0x7f00100af020 con 0x7f00100b3970 2024-03-29T11:00:35.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:35.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:35.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:35.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:00:35.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 1 -- 172.21.15.71:0/2450517603 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00100b0bf0 con 0x7f00100b3970 2024-03-29T11:00:35.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 1 -- 172.21.15.71:0/2450517603 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 829083870 0 0) 0x7f00100b0910 con 0x7f001013f390 2024-03-29T11:00:35.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:35.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:35.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:35.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 10 monclient(hunting): my global_id is 4186 2024-03-29T11:00:35.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 1 -- 172.21.15.71:0/2450517603 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00100af020 con 0x7f001013f390 2024-03-29T11:00:35.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 1 -- 172.21.15.71:0/2450517603 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3729879450 0 0) 0x7efffc003520 con 0x7f00100b4710 2024-03-29T11:00:35.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 1 -- 172.21.15.71:0/2450517603 >> v1:172.21.15.137:6789/0 conn(0x7f001013f390 legacy=0x7f001013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:35.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 1 -- 172.21.15.71:0/2450517603 >> v1:172.21.15.71:6789/0 conn(0x7f00100b3970 legacy=0x7f00100af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:35.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 10 monclient: _finish_hunting 0 2024-03-29T11:00:35.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 1 monclient: found mon.noname-b 2024-03-29T11:00:35.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:35.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:00:35.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 1 -- 172.21.15.71:0/2450517603 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00100aeb00 con 0x7f00100b4710 2024-03-29T11:00:35.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 10 monclient: _finish_auth 0 2024-03-29T11:00:35.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 10 monclient: _check_auth_tickets 2024-03-29T11:00:35.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:35.527 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 1 -- 172.21.15.71:0/2450517603 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f0000001690 con 0x7f00100b4710 2024-03-29T11:00:35.527 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f0016214640 5 monclient: authenticate success, global_id 4190 2024-03-29T11:00:35.527 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f0016214640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:35.527 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:35.527 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:35.527 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.534+0000 7f000effd640 10 monclient: dump: 2024-03-29T11:00:35.527 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:00:35.527 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:35.527 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:35.527 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:35.527 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:35.527 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:35.527 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:35.527 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:35.528 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:35.528 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:00:35.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.535+0000 7f0016214640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:35.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.535+0000 7f000effd640 1 -- 172.21.15.71:0/2450517603 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00000017f0 con 0x7f00100b4710 2024-03-29T11:00:35.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.535+0000 7f000effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:35.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.535+0000 7f000effd640 1 -- 172.21.15.71:0/2450517603 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f0000001c40 con 0x7f00100b4710 2024-03-29T11:00:35.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.535+0000 7f0016214640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:35.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.535+0000 7f0016214640 4 set_mon_vals no callback set 2024-03-29T11:00:35.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.535+0000 7f000effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:35.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.535+0000 7f000effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:35.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.535+0000 7f000effd640 10 monclient: dump: 2024-03-29T11:00:35.528 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:00:35.528 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:35.528 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:35.529 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:35.529 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:35.529 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:35.529 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:35.529 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:35.529 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:35.529 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:00:35.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.535+0000 7f0016214640 10 monclient: shutdown 2024-03-29T11:00:35.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.535+0000 7f0016214640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:35.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.535+0000 7f0016214640 1 -- 172.21.15.71:0/2450517603 >> v1:172.21.15.71:6790/0 conn(0x7f00100b4710 legacy=0x7f00100b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:35.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.535+0000 7f0016214640 1 -- 172.21.15.71:0/2450517603 shutdown_connections 2024-03-29T11:00:35.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.535+0000 7f0016214640 1 -- 172.21.15.71:0/2450517603 >> 172.21.15.71:0/2450517603 conn(0x7f0010066a00 msgr2=0x7f0010066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:35.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.535+0000 7f0016214640 1 -- 172.21.15.71:0/2450517603 shutdown_connections 2024-03-29T11:00:35.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.535+0000 7f0016214640 1 -- 172.21.15.71:0/2450517603 wait complete. 2024-03-29T11:00:35.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.536+0000 7f0016214640 10 monclient: build_initial_monmap 2024-03-29T11:00:35.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.536+0000 7f0016214640 10 monclient: monmap: 2024-03-29T11:00:35.530 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:00:35.530 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:35.530 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:00:35.530 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:00:35.530 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:35.530 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:35.530 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:35.530 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:00:35.530 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:00:35.530 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:00:35.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.536+0000 7f0016214640 1 librados: starting msgr at 2024-03-29T11:00:35.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.536+0000 7f0016214640 1 librados: starting objecter 2024-03-29T11:00:35.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.536+0000 7f0016214640 1 Processor -- start 2024-03-29T11:00:35.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.536+0000 7f0016214640 1 -- start start 2024-03-29T11:00:35.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.536+0000 7f0016214640 1 librados: setting wanted keys 2024-03-29T11:00:35.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.536+0000 7f0016214640 1 librados: calling monclient init 2024-03-29T11:00:35.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.536+0000 7f0016214640 10 monclient: init 2024-03-29T11:00:35.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f0016214640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:35.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f0016214640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:00:35.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f0016214640 10 monclient(hunting): picked mon.noname-b con 0x7f00100b3970 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:35.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f0016214640 10 monclient(hunting): picked mon.noname-c con 0x7f00100b4710 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:35.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f0016214640 10 monclient(hunting): picked mon.noname-a con 0x7f001013f390 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:35.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f0016214640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0010135720 con 0x7f001013f390 2024-03-29T11:00:35.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f0016214640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0010136d90 con 0x7f00100b3970 2024-03-29T11:00:35.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f0016214640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0010137070 con 0x7f00100b4710 2024-03-29T11:00:35.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f0016214640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:35.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f0016214640 10 monclient(hunting): authenticate will time out at 791.796418s 2024-03-29T11:00:35.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f0006ffd640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f00100b4710 0x7f0010135bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:38342/0 (socket says 172.21.15.71:38342) 2024-03-29T11:00:35.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000ffff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f001013f390 0x7f00101e1e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:60426/0 (socket says 172.21.15.71:60426) 2024-03-29T11:00:35.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f0006ffd640 1 -- 172.21.15.71:0/866213955 learned_addr learned my addr 172.21.15.71:0/866213955 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:00:35.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 client.?.objecter ms_handle_connect 0x7f00100b4710 2024-03-29T11:00:35.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:35.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 client.?.objecter ms_handle_connect 0x7f001013f390 2024-03-29T11:00:35.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:35.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 client.?.objecter ms_handle_connect 0x7f00100b3970 2024-03-29T11:00:35.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:35.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 1 -- 172.21.15.71:0/866213955 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794937529 0 0) 0x7f0010137070 con 0x7f00100b4710 2024-03-29T11:00:35.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:00:35.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:35.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:35.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 monclient(hunting): my global_id is 4193 2024-03-29T11:00:35.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 1 -- 172.21.15.71:0/866213955 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efff0004eb0 con 0x7f00100b4710 2024-03-29T11:00:35.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 1 -- 172.21.15.71:0/866213955 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1032702269 0 0) 0x7f0010135720 con 0x7f001013f390 2024-03-29T11:00:35.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:00:35.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:35.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:35.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 monclient(hunting): my global_id is 4194 2024-03-29T11:00:35.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 1 -- 172.21.15.71:0/866213955 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0010137070 con 0x7f001013f390 2024-03-29T11:00:35.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 1 -- 172.21.15.71:0/866213955 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3769356702 0 0) 0x7f0010136d90 con 0x7f00100b3970 2024-03-29T11:00:35.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:00:35.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:35.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:35.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 monclient(hunting): my global_id is 4189 2024-03-29T11:00:35.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 1 -- 172.21.15.71:0/866213955 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0010135720 con 0x7f00100b3970 2024-03-29T11:00:35.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 1 -- 172.21.15.71:0/866213955 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3450167102 0 0) 0x7f0010137070 con 0x7f001013f390 2024-03-29T11:00:35.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 1 -- 172.21.15.71:0/866213955 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0010136d90 con 0x7f001013f390 2024-03-29T11:00:35.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 1 -- 172.21.15.71:0/866213955 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f0008001b50 con 0x7f001013f390 2024-03-29T11:00:35.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:35.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:00:35.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 monclient(hunting): dump: 2024-03-29T11:00:35.535 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:00:35.535 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:35.535 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:35.535 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:35.535 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:35.535 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:35.535 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:35.535 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:35.535 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:35.536 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:00:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 1 -- 172.21.15.71:0/866213955 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2453960051 0 0) 0x7efff0004eb0 con 0x7f00100b4710 2024-03-29T11:00:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 1 -- 172.21.15.71:0/866213955 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0010137070 con 0x7f00100b4710 2024-03-29T11:00:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 1 -- 172.21.15.71:0/866213955 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2742733979 0 0) 0x7f0010135720 con 0x7f00100b3970 2024-03-29T11:00:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 1 -- 172.21.15.71:0/866213955 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efff0004eb0 con 0x7f00100b3970 2024-03-29T11:00:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 1 -- 172.21.15.71:0/866213955 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7efff8001a50 con 0x7f00100b4710 2024-03-29T11:00:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 monclient(hunting): dump: 2024-03-29T11:00:35.536 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:00:35.536 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:35.536 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:35.537 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:35.537 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:35.537 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:35.537 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:35.537 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:35.537 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:35.537 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:00:35.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 1 -- 172.21.15.71:0/866213955 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f0000001110 con 0x7f00100b3970 2024-03-29T11:00:35.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:35.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:35.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.537+0000 7f000cff9640 10 monclient(hunting): dump: 2024-03-29T11:00:35.537 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:00:35.538 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:35.538 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:35.538 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:35.538 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:35.538 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:35.538 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:35.538 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:35.538 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:35.538 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:00:35.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f000cff9640 1 -- 172.21.15.71:0/866213955 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3234104012 0 0) 0x7f0010136d90 con 0x7f001013f390 2024-03-29T11:00:35.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f000cff9640 1 -- 172.21.15.71:0/866213955 >> v1:172.21.15.71:6790/0 conn(0x7f00100b4710 legacy=0x7f0010135bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:35.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f000cff9640 1 -- 172.21.15.71:0/866213955 >> v1:172.21.15.137:6789/0 conn(0x7f00100b3970 legacy=0x7f0010137770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:35.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f000cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:00:35.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f000cff9640 1 monclient: found mon.a 2024-03-29T11:00:35.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f000cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:35.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f000cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:00:35.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f000cff9640 1 -- 172.21.15.71:0/866213955 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0010137350 con 0x7f001013f390 2024-03-29T11:00:35.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f000cff9640 10 monclient: _finish_auth 0 2024-03-29T11:00:35.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f000cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:00:35.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f000cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:35.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f0016214640 5 monclient: authenticate success, global_id 4194 2024-03-29T11:00:35.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f0016214640 10 monclient: _renew_subs 2024-03-29T11:00:35.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f0016214640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:00:35.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f0016214640 1 -- 172.21.15.71:0/866213955 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0010134c80 con 0x7f001013f390 2024-03-29T11:00:35.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f0016214640 10 client.4194.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:35.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f0016214640 10 monclient: _renew_subs 2024-03-29T11:00:35.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f0016214640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:00:35.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f0016214640 1 -- 172.21.15.71:0/866213955 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0010142010 con 0x7f001013f390 2024-03-29T11:00:35.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f0016214640 1 librados: init done 2024-03-29T11:00:35.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f000cff9640 1 -- 172.21.15.71:0/866213955 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00080020c0 con 0x7f001013f390 2024-03-29T11:00:35.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f000cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:35.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f000cff9640 1 -- 172.21.15.71:0/866213955 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f0008003b30 con 0x7f001013f390 2024-03-29T11:00:35.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f000cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:35.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f000cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:35.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f000e7fc640 4 set_mon_vals no callback set 2024-03-29T11:00:35.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f000cff9640 10 monclient: dump: 2024-03-29T11:00:35.541 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:00:35.541 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:35.541 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:35.541 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:35.541 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:35.541 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:35.541 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:35.542 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:35.542 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:35.542 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:00:35.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f000cff9640 1 -- 172.21.15.71:0/866213955 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f00080040e0 con 0x7f001013f390 2024-03-29T11:00:35.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f000cff9640 1 -- 172.21.15.71:0/866213955 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f0008006c00 con 0x7f001013f390 2024-03-29T11:00:35.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f000cff9640 10 client.4194.objecter ms_dispatch 0x7f0010000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:35.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f000cff9640 3 client.4194.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:35.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f000cff9640 3 client.4194.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:35.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f000cff9640 20 client.4194.objecter dump_active .. 0 homeless 2024-03-29T11:00:35.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f0016214640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:35.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f0016214640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:35.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.538+0000 7f0016214640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:00:35.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.539+0000 7f0016214640 1 -- 172.21.15.71:0/866213955 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efff4005080 con 0x7f001013f390 2024-03-29T11:00:35.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.541+0000 7f000cff9640 1 -- 172.21.15.71:0/866213955 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0008004680 con 0x7f001013f390 2024-03-29T11:00:35.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.541+0000 7f000cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:35.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.541+0000 7f000cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:35.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.602+0000 7f0016214640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:35.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.602+0000 7f0016214640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:35.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.602+0000 7f0016214640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:00:35.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.602+0000 7f0016214640 1 -- 172.21.15.71:0/866213955 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efff4005330 con 0x7f001013f390 2024-03-29T11:00:35.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.602+0000 7f000cff9640 1 -- 172.21.15.71:0/866213955 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f0008005b40 con 0x7f001013f390 2024-03-29T11:00:35.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.602+0000 7f000cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:35.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.602+0000 7f000cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:35.584 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:00:35.585 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:35.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.603+0000 7f0016214640 10 librados: watch_flush enter 2024-03-29T11:00:35.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.603+0000 7f0016214640 10 librados: watch_flush exit 2024-03-29T11:00:35.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.603+0000 7f0016214640 20 client.4194.objecter shutdown clearing up homeless session... 2024-03-29T11:00:35.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.603+0000 7f0016214640 10 client.4194.objecter successfully canceled tick 2024-03-29T11:00:35.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.603+0000 7f0016214640 10 monclient: shutdown 2024-03-29T11:00:35.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.603+0000 7f0016214640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:35.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.603+0000 7f0016214640 1 -- 172.21.15.71:0/866213955 >> v1:172.21.15.71:6789/0 conn(0x7f001013f390 legacy=0x7f00101e1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:35.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.603+0000 7f0016214640 1 -- 172.21.15.71:0/866213955 shutdown_connections 2024-03-29T11:00:35.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.603+0000 7f0016214640 1 -- 172.21.15.71:0/866213955 >> 172.21.15.71:0/866213955 conn(0x7f0010066a00 msgr2=0x7f0010069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:35.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.603+0000 7f0016214640 1 -- 172.21.15.71:0/866213955 shutdown_connections 2024-03-29T11:00:35.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.603+0000 7f0016214640 1 -- 172.21.15.71:0/866213955 wait complete. 2024-03-29T11:00:35.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:35.604+0000 7f0016214640 1 librados: shutdown 2024-03-29T11:00:35.593 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:35.874 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:00:35.893+0000 7f5976230040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:00:35.875 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T11:00:35.895+0000 7fab49e36040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:00:36.146 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T11:00:36.166+0000 7fab49e36040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:00:36.151 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:00:36.170+0000 7f5976230040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:00:36.196 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T11:00:36.216+0000 7fab49e36040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:00:36.202 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:00:36.222+0000 7f5976230040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:00:36.308 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T11:00:36.328+0000 7fab49e36040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:00:36.318 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:00:36.337+0000 7f5976230040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:00:37.321 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T11:00:37.341+0000 7faaeaae9640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:00:37.321 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-03-29T11:00:37.341+0000 7faaeaae9640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:00:38.436 INFO:tasks.ceph.osd.1.smithi071.stderr:2024-03-29T11:00:38.455+0000 7f09baa0d640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:00:38.487 INFO:tasks.ceph.osd.0.smithi071.stderr:2024-03-29T11:00:38.507+0000 7f35e0615640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:00:38.494 INFO:tasks.ceph.osd.7.smithi137.stderr:2024-03-29T11:00:38.514+0000 7f7fe9149640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:00:38.535 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-03-29T11:00:38.554+0000 7f5bda345640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:00:38.540 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-03-29T11:00:38.559+0000 7f7e00611640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:00:38.541 INFO:tasks.ceph.osd.5.smithi137.stderr:2024-03-29T11:00:38.561+0000 7f062b023640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:00:38.544 INFO:tasks.ceph.osd.6.smithi137.stderr:2024-03-29T11:00:38.564+0000 7f322cc17640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:00:38.547 INFO:tasks.ceph.osd.4.smithi137.stderr:2024-03-29T11:00:38.567+0000 7f22cba1c640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:00:42.400 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:42.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.576+0000 7fe7088a8640 10 monclient: get_monmap_and_config 2024-03-29T11:00:42.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.576+0000 7fe7088a8640 10 monclient: build_initial_monmap 2024-03-29T11:00:42.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.576+0000 7fe7088a8640 10 monclient: monmap: 2024-03-29T11:00:42.557 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:00:42.557 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:42.558 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:00:42.577586+0000 2024-03-29T11:00:42.558 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:00:42.577586+0000 2024-03-29T11:00:42.558 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:42.558 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:42.558 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:42.558 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:00:42.558 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:00:42.558 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:00:42.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.577+0000 7fe7088a8640 1 Processor -- start 2024-03-29T11:00:42.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.577+0000 7fe7088a8640 1 -- start start 2024-03-29T11:00:42.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.577+0000 7fe7088a8640 10 monclient: init 2024-03-29T11:00:42.560 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.579+0000 7fe7088a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:42.560 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.579+0000 7fe7088a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:00:42.560 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.579+0000 7fe7088a8640 10 monclient(hunting): picked mon.noname-a con 0x7fe7041422b0 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:42.560 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.579+0000 7fe7088a8640 10 monclient(hunting): picked mon.noname-b con 0x7fe704142690 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:42.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.579+0000 7fe7088a8640 10 monclient(hunting): picked mon.noname-c con 0x7fe7040b17d0 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:42.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.579+0000 7fe7088a8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe704142c80 con 0x7fe7041422b0 2024-03-29T11:00:42.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.579+0000 7fe7088a8640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7040aae40 con 0x7fe7040b17d0 2024-03-29T11:00:42.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.579+0000 7fe702d9d640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fe7040b17d0 0x7fe7040b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:40150/0 (socket says 172.21.15.71:40150) 2024-03-29T11:00:42.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.579+0000 7fe702d9d640 1 -- 172.21.15.71:0/1387894566 learned_addr learned my addr 172.21.15.71:0/1387894566 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:00:42.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.580+0000 7fe7088a8640 1 -- 172.21.15.71:0/1387894566 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe70413fbd0 con 0x7fe704142690 2024-03-29T11:00:42.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.580+0000 7fe7088a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:42.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.580+0000 7fe7088a8640 10 monclient(hunting): authenticate will time out at 798.838266s 2024-03-29T11:00:42.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.580+0000 7fe70159a640 1 -- 172.21.15.71:0/1387894566 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197038043 0 0) 0x7fe70413fbd0 con 0x7fe704142690 2024-03-29T11:00:42.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.580+0000 7fe70159a640 10 monclient(hunting): _negotiate 2024-03-29T11:00:42.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.580+0000 7fe70159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:42.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.580+0000 7fe70159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:42.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.580+0000 7fe70159a640 10 monclient(hunting): my global_id is 4238 2024-03-29T11:00:42.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.580+0000 7fe70159a640 1 -- 172.21.15.71:0/1387894566 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6f0003520 con 0x7fe704142690 2024-03-29T11:00:42.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.581+0000 7fe70159a640 1 -- 172.21.15.71:0/1387894566 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2425025234 0 0) 0x7fe7040aae40 con 0x7fe7040b17d0 2024-03-29T11:00:42.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.581+0000 7fe70159a640 10 monclient(hunting): _negotiate 2024-03-29T11:00:42.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.581+0000 7fe70159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:42.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.581+0000 7fe70159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:42.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.581+0000 7fe70159a640 10 monclient(hunting): my global_id is 4228 2024-03-29T11:00:42.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.581+0000 7fe70159a640 1 -- 172.21.15.71:0/1387894566 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6f0003840 con 0x7fe7040b17d0 2024-03-29T11:00:42.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.581+0000 7fe70159a640 1 -- 172.21.15.71:0/1387894566 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3389388757 0 0) 0x7fe6f0003840 con 0x7fe7040b17d0 2024-03-29T11:00:42.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.581+0000 7fe70159a640 1 -- 172.21.15.71:0/1387894566 >> v1:172.21.15.71:6790/0 conn(0x7fe704142690 legacy=0x7fe7040b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:42.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.581+0000 7fe70159a640 1 -- 172.21.15.71:0/1387894566 >> v1:172.21.15.71:6789/0 conn(0x7fe7041422b0 legacy=0x7fe70413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:42.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.581+0000 7fe70159a640 10 monclient: _finish_hunting 0 2024-03-29T11:00:42.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.581+0000 7fe70159a640 1 monclient: found mon.noname-c 2024-03-29T11:00:42.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.581+0000 7fe70159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:42.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.581+0000 7fe70159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.137:6789/0 2024-03-29T11:00:42.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.581+0000 7fe70159a640 1 -- 172.21.15.71:0/1387894566 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe704151940 con 0x7fe7040b17d0 2024-03-29T11:00:42.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.582+0000 7fe70159a640 10 monclient: _finish_auth 0 2024-03-29T11:00:42.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.582+0000 7fe70159a640 10 monclient: _check_auth_tickets 2024-03-29T11:00:42.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.582+0000 7fe70159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:42.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.582+0000 7fe70159a640 1 -- 172.21.15.71:0/1387894566 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fe6fc001670 con 0x7fe7040b17d0 2024-03-29T11:00:42.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.582+0000 7fe7088a8640 5 monclient: authenticate success, global_id 4228 2024-03-29T11:00:42.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.582+0000 7fe70159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:42.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.582+0000 7fe70159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:42.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.582+0000 7fe70159a640 10 monclient: dump: 2024-03-29T11:00:42.565 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:00:42.565 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:42.565 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:42.565 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:42.565 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:42.565 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:42.565 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:42.565 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:42.565 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:42.565 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:00:42.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.582+0000 7fe7088a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:42.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.583+0000 7fe70159a640 1 -- 172.21.15.71:0/1387894566 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6fc0017d0 con 0x7fe7040b17d0 2024-03-29T11:00:42.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.583+0000 7fe70159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:42.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.583+0000 7fe7088a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:42.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.583+0000 7fe7088a8640 4 set_mon_vals no callback set 2024-03-29T11:00:42.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.583+0000 7fe70159a640 1 -- 172.21.15.71:0/1387894566 <== mon.1 v1:172.21.15.137:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fe6fc001ba0 con 0x7fe7040b17d0 2024-03-29T11:00:42.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.583+0000 7fe70159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:42.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.583+0000 7fe70159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:42.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.583+0000 7fe70159a640 10 monclient: dump: 2024-03-29T11:00:42.566 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:00:42.566 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:42.566 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:42.566 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:42.566 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:42.566 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:42.566 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:42.566 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:42.566 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:42.567 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:00:42.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.583+0000 7fe7088a8640 10 monclient: shutdown 2024-03-29T11:00:42.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.584+0000 7fe7088a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:42.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.584+0000 7fe7088a8640 1 -- 172.21.15.71:0/1387894566 >> v1:172.21.15.137:6789/0 conn(0x7fe7040b17d0 legacy=0x7fe7040b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:42.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.584+0000 7fe7088a8640 1 -- 172.21.15.71:0/1387894566 shutdown_connections 2024-03-29T11:00:42.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.584+0000 7fe7088a8640 1 -- 172.21.15.71:0/1387894566 >> 172.21.15.71:0/1387894566 conn(0x7fe704066a00 msgr2=0x7fe704066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:42.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.584+0000 7fe7088a8640 1 -- 172.21.15.71:0/1387894566 shutdown_connections 2024-03-29T11:00:42.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.585+0000 7fe7088a8640 1 -- 172.21.15.71:0/1387894566 wait complete. 2024-03-29T11:00:42.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.586+0000 7fe7088a8640 10 monclient: build_initial_monmap 2024-03-29T11:00:42.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.587+0000 7fe7088a8640 10 monclient: monmap: 2024-03-29T11:00:42.569 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:00:42.569 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:42.569 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:00:42.569 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:00:42.569 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:42.569 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:42.569 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:42.569 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:00:42.569 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:00:42.569 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:00:42.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.587+0000 7fe7088a8640 1 librados: starting msgr at 2024-03-29T11:00:42.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.588+0000 7fe7088a8640 1 librados: starting objecter 2024-03-29T11:00:42.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.588+0000 7fe7088a8640 1 Processor -- start 2024-03-29T11:00:42.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.588+0000 7fe7088a8640 1 -- start start 2024-03-29T11:00:42.577 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.588+0000 7fe7088a8640 1 librados: setting wanted keys 2024-03-29T11:00:42.577 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.588+0000 7fe7088a8640 1 librados: calling monclient init 2024-03-29T11:00:42.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.588+0000 7fe7088a8640 10 monclient: init 2024-03-29T11:00:42.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.589+0000 7fe7088a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:42.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.589+0000 7fe7088a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:00:42.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.589+0000 7fe7088a8640 10 monclient(hunting): picked mon.noname-c con 0x7fe7040b17d0 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:42.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.589+0000 7fe7088a8640 10 monclient(hunting): picked mon.noname-a con 0x7fe7041422b0 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:42.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.589+0000 7fe7088a8640 10 monclient(hunting): picked mon.noname-b con 0x7fe704142690 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:42.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.589+0000 7fe7088a8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7040b03b0 con 0x7fe7041422b0 2024-03-29T11:00:42.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.589+0000 7fe7088a8640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7040b0fa0 con 0x7fe704142690 2024-03-29T11:00:42.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.589+0000 7fe7088a8640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7040b1280 con 0x7fe7040b17d0 2024-03-29T11:00:42.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.589+0000 7fe7088a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:42.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.589+0000 7fe7088a8640 10 monclient(hunting): authenticate will time out at 798.847688s 2024-03-29T11:00:42.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.589+0000 7fe70259c640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fe7040b17d0 0x7fe7040af170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:34194/0 (socket says 172.21.15.71:34194) 2024-03-29T11:00:42.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.589+0000 7fe70259c640 1 -- 172.21.15.71:0/437990393 learned_addr learned my addr 172.21.15.71:0/437990393 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:00:42.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.589+0000 7fe6faffd640 10 client.?.objecter ms_handle_connect 0x7fe7041422b0 2024-03-29T11:00:42.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.590+0000 7fe6faffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:42.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.590+0000 7fe6faffd640 10 client.?.objecter ms_handle_connect 0x7fe7040b17d0 2024-03-29T11:00:42.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.590+0000 7fe6faffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:42.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.590+0000 7fe6faffd640 10 client.?.objecter ms_handle_connect 0x7fe704142690 2024-03-29T11:00:42.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.590+0000 7fe6faffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:42.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.590+0000 7fe6faffd640 1 -- 172.21.15.71:0/437990393 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2829638057 0 0) 0x7fe7040b03b0 con 0x7fe7041422b0 2024-03-29T11:00:42.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.590+0000 7fe6faffd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:42.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.590+0000 7fe6faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:42.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.590+0000 7fe6faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:42.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.590+0000 7fe6faffd640 10 monclient(hunting): my global_id is 4242 2024-03-29T11:00:42.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.590+0000 7fe6faffd640 1 -- 172.21.15.71:0/437990393 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6e4004eb0 con 0x7fe7041422b0 2024-03-29T11:00:42.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.590+0000 7fe6faffd640 1 -- 172.21.15.71:0/437990393 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3859598544 0 0) 0x7fe7040b0fa0 con 0x7fe704142690 2024-03-29T11:00:42.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.591+0000 7fe6faffd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:42.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.591+0000 7fe6faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:42.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.591+0000 7fe6faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:42.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.591+0000 7fe6faffd640 10 monclient(hunting): my global_id is 4231 2024-03-29T11:00:42.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.591+0000 7fe6faffd640 1 -- 172.21.15.71:0/437990393 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7040b03b0 con 0x7fe704142690 2024-03-29T11:00:42.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.591+0000 7fe6faffd640 1 -- 172.21.15.71:0/437990393 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161179786 0 0) 0x7fe7040b1280 con 0x7fe7040b17d0 2024-03-29T11:00:42.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.591+0000 7fe6faffd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:42.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.591+0000 7fe6faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:42.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.591+0000 7fe6faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:42.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.591+0000 7fe6faffd640 10 monclient(hunting): my global_id is 4241 2024-03-29T11:00:42.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.591+0000 7fe6faffd640 1 -- 172.21.15.71:0/437990393 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7040b0fa0 con 0x7fe7040b17d0 2024-03-29T11:00:42.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.591+0000 7fe6faffd640 1 -- 172.21.15.71:0/437990393 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1350200376 0 0) 0x7fe7040b03b0 con 0x7fe704142690 2024-03-29T11:00:42.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.591+0000 7fe6faffd640 1 -- 172.21.15.71:0/437990393 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7040b1280 con 0x7fe704142690 2024-03-29T11:00:42.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.591+0000 7fe6faffd640 1 -- 172.21.15.71:0/437990393 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fe6fc001ba0 con 0x7fe704142690 2024-03-29T11:00:42.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.591+0000 7fe6faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:42.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.591+0000 7fe6faffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:42.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.591+0000 7fe6faffd640 10 monclient(hunting): dump: 2024-03-29T11:00:42.581 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:00:42.581 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:42.581 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:42.581 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:42.581 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:42.581 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:42.581 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:42.581 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:42.581 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:42.581 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:00:42.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.591+0000 7fe6faffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:42.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.591+0000 7fe6faffd640 1 -- 172.21.15.71:0/437990393 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1570982121 0 0) 0x7fe6e4004eb0 con 0x7fe7041422b0 2024-03-29T11:00:42.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.591+0000 7fe6faffd640 1 -- 172.21.15.71:0/437990393 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7040b03b0 con 0x7fe7041422b0 2024-03-29T11:00:42.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.591+0000 7fe6faffd640 1 -- 172.21.15.71:0/437990393 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fe6ec001810 con 0x7fe7041422b0 2024-03-29T11:00:42.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe6faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:42.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe6faffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:42.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe6faffd640 10 monclient(hunting): dump: 2024-03-29T11:00:42.582 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:00:42.582 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:42.582 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:42.582 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:42.582 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:42.582 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:42.583 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:42.583 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:42.583 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:42.583 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:00:42.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe6faffd640 1 -- 172.21.15.71:0/437990393 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3933162109 0 0) 0x7fe7040b1280 con 0x7fe704142690 2024-03-29T11:00:42.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe6faffd640 1 -- 172.21.15.71:0/437990393 >> v1:172.21.15.71:6790/0 conn(0x7fe7040b17d0 legacy=0x7fe7040af170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:42.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe6faffd640 1 -- 172.21.15.71:0/437990393 >> v1:172.21.15.71:6789/0 conn(0x7fe7041422b0 legacy=0x7fe7041daad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:42.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe6faffd640 10 monclient: _finish_hunting 0 2024-03-29T11:00:42.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe6faffd640 1 monclient: found mon.b 2024-03-29T11:00:42.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe6faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:42.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe6faffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:42.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe6faffd640 1 -- 172.21.15.71:0/437990393 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7040afe10 con 0x7fe704142690 2024-03-29T11:00:42.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe6faffd640 10 monclient: _finish_auth 0 2024-03-29T11:00:42.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe6faffd640 10 monclient: _check_auth_tickets 2024-03-29T11:00:42.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe6faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:42.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe7088a8640 5 monclient: authenticate success, global_id 4231 2024-03-29T11:00:42.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe7088a8640 10 monclient: _renew_subs 2024-03-29T11:00:42.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe7088a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:42.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe7088a8640 1 -- 172.21.15.71:0/437990393 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7040b00c0 con 0x7fe704142690 2024-03-29T11:00:42.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe7088a8640 10 client.4231.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:42.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe7088a8640 10 monclient: _renew_subs 2024-03-29T11:00:42.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe7088a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:42.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe7088a8640 1 -- 172.21.15.71:0/437990393 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7041e2fc0 con 0x7fe704142690 2024-03-29T11:00:42.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe7088a8640 1 librados: init done 2024-03-29T11:00:42.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe6faffd640 1 -- 172.21.15.71:0/437990393 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6fc004a90 con 0x7fe704142690 2024-03-29T11:00:42.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe6faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:42.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe6fbfff640 4 set_mon_vals no callback set 2024-03-29T11:00:42.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe6faffd640 1 -- 172.21.15.71:0/437990393 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fe6fc000ed0 con 0x7fe704142690 2024-03-29T11:00:42.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe6faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:42.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe6faffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:42.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.592+0000 7fe6faffd640 10 monclient: dump: 2024-03-29T11:00:42.585 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:00:42.585 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:42.585 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:42.585 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:42.585 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:42.585 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:42.585 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:42.585 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:42.585 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:42.585 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:00:42.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.595+0000 7fe6faffd640 1 -- 172.21.15.71:0/437990393 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7fe6fc01ac10 con 0x7fe704142690 2024-03-29T11:00:42.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.595+0000 7fe7088a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:42.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.595+0000 7fe7088a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:42.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.595+0000 7fe7088a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:42.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.595+0000 7fe7088a8640 1 -- 172.21.15.71:0/437990393 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e8005080 con 0x7fe704142690 2024-03-29T11:00:42.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.595+0000 7fe70259c640 10 monclient: get_auth_request con 0x7fe6e406df90 auth_method 0 2024-03-29T11:00:42.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.595+0000 7fe6faffd640 1 -- 172.21.15.71:0/437990393 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7fe6fc0849f0 con 0x7fe704142690 2024-03-29T11:00:42.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.595+0000 7fe6faffd640 10 client.4231.objecter ms_dispatch 0x7fe704000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:42.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.595+0000 7fe6faffd640 3 client.4231.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:42.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.595+0000 7fe6faffd640 3 client.4231.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:42.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.596+0000 7fe6faffd640 20 client.4231.objecter dump_active .. 0 homeless 2024-03-29T11:00:42.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.596+0000 7fe6faffd640 10 client.4231.objecter ms_handle_connect 0x7fe6e406df90 2024-03-29T11:00:42.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.602+0000 7fe6faffd640 1 -- 172.21.15.71:0/437990393 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe6fc0502d0 con 0x7fe704142690 2024-03-29T11:00:42.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.602+0000 7fe6faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:42.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.602+0000 7fe6faffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:42.973 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.993+0000 7fe7088a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:42.973 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.993+0000 7fe7088a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:42.973 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.993+0000 7fe7088a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:42.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.993+0000 7fe7088a8640 1 -- 172.21.15.71:0/437990393 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6e8005330 con 0x7fe704142690 2024-03-29T11:00:42.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.994+0000 7fe6faffd640 1 -- 172.21.15.71:0/437990393 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7fe6fc052200 con 0x7fe704142690 2024-03-29T11:00:42.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.995+0000 7fe6faffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:42.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.995+0000 7fe6faffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:42.976 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:00:42.976 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:42.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.998+0000 7fe7088a8640 10 librados: watch_flush enter 2024-03-29T11:00:42.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.998+0000 7fe7088a8640 10 librados: watch_flush exit 2024-03-29T11:00:42.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.998+0000 7fe7088a8640 20 client.4231.objecter shutdown clearing up homeless session... 2024-03-29T11:00:42.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.998+0000 7fe7088a8640 10 client.4231.objecter successfully canceled tick 2024-03-29T11:00:42.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.998+0000 7fe7088a8640 1 -- 172.21.15.71:0/437990393 >> v1:172.21.15.137:6816/973332665 conn(0x7fe6e406df90 legacy=0x7fe6e4070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:42.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.998+0000 7fe7088a8640 10 monclient: shutdown 2024-03-29T11:00:42.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.998+0000 7fe7088a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:42.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.998+0000 7fe7088a8640 1 -- 172.21.15.71:0/437990393 >> v1:172.21.15.137:6789/0 conn(0x7fe704142690 legacy=0x7fe7041db1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:42.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.998+0000 7fe7088a8640 1 -- 172.21.15.71:0/437990393 shutdown_connections 2024-03-29T11:00:42.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.998+0000 7fe7088a8640 1 -- 172.21.15.71:0/437990393 >> 172.21.15.71:0/437990393 conn(0x7fe704066a00 msgr2=0x7fe70413d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:42.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.999+0000 7fe7088a8640 1 -- 172.21.15.71:0/437990393 shutdown_connections 2024-03-29T11:00:42.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.999+0000 7fe7088a8640 1 -- 172.21.15.71:0/437990393 wait complete. 2024-03-29T11:00:42.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:42.999+0000 7fe7088a8640 1 librados: shutdown 2024-03-29T11:00:42.994 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:49.801 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:49.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.954+0000 7f62cc1a3640 10 monclient: get_monmap_and_config 2024-03-29T11:00:49.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.954+0000 7f62cc1a3640 10 monclient: build_initial_monmap 2024-03-29T11:00:49.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.954+0000 7f62cc1a3640 10 monclient: monmap: 2024-03-29T11:00:49.936 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:00:49.936 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:49.936 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:00:49.955675+0000 2024-03-29T11:00:49.936 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:00:49.955675+0000 2024-03-29T11:00:49.936 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:49.936 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:49.936 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:49.936 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:00:49.936 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:00:49.936 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:00:49.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.955+0000 7f62cc1a3640 1 Processor -- start 2024-03-29T11:00:49.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.955+0000 7f62cc1a3640 1 -- start start 2024-03-29T11:00:49.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.955+0000 7f62cc1a3640 10 monclient: init 2024-03-29T11:00:49.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.956+0000 7f62cc1a3640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:49.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.956+0000 7f62cc1a3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:00:49.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.956+0000 7f62cc1a3640 10 monclient(hunting): picked mon.noname-b con 0x7f62c40aa430 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:49.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.956+0000 7f62cc1a3640 10 monclient(hunting): picked mon.noname-c con 0x7f62c40a9930 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:49.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.956+0000 7f62cc1a3640 10 monclient(hunting): picked mon.noname-a con 0x7f62c40ae6f0 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:49.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.956+0000 7f62cc1a3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c40ab140 con 0x7f62c40ae6f0 2024-03-29T11:00:49.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.956+0000 7f62cc1a3640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c40ab700 con 0x7f62c40a9930 2024-03-29T11:00:49.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.956+0000 7f62cc1a3640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c413eeb0 con 0x7f62c40aa430 2024-03-29T11:00:49.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.956+0000 7f62cc1a3640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:49.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.956+0000 7f62cc1a3640 10 monclient(hunting): authenticate will time out at 806.214830s 2024-03-29T11:00:49.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.956+0000 7f62ca740640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f62c40ae6f0 0x7f62c414b0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:57204/0 (socket says 172.21.15.71:57204) 2024-03-29T11:00:49.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.956+0000 7f62ca740640 1 -- 172.21.15.71:0/2833172973 learned_addr learned my addr 172.21.15.71:0/2833172973 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:00:49.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.957+0000 7f62c8f3d640 1 -- 172.21.15.71:0/2833172973 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465575272 0 0) 0x7f62c40ab140 con 0x7f62c40ae6f0 2024-03-29T11:00:49.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.957+0000 7f62c8f3d640 10 monclient(hunting): _negotiate 2024-03-29T11:00:49.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.957+0000 7f62c8f3d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:49.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.957+0000 7f62c8f3d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:49.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.957+0000 7f62c8f3d640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:00:49.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.957+0000 7f62c8f3d640 1 -- 172.21.15.71:0/2833172973 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62b0003520 con 0x7f62c40ae6f0 2024-03-29T11:00:49.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.957+0000 7f62c8f3d640 1 -- 172.21.15.71:0/2833172973 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2341140338 0 0) 0x7f62b0003520 con 0x7f62c40ae6f0 2024-03-29T11:00:49.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.957+0000 7f62c8f3d640 1 -- 172.21.15.71:0/2833172973 >> v1:172.21.15.71:6790/0 conn(0x7f62c40aa430 legacy=0x7f62c413b9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:00:49.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.959+0000 7f62c8f3d640 1 -- 172.21.15.71:0/2833172973 >> v1:172.21.15.137:6789/0 conn(0x7f62c40a9930 legacy=0x7f62c40adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:49.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.959+0000 7f62c8f3d640 10 monclient: _finish_hunting 0 2024-03-29T11:00:49.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.959+0000 7f62c8f3d640 1 monclient: found mon.noname-a 2024-03-29T11:00:49.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.960+0000 7f62c8f3d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:49.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.960+0000 7f62c8f3d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:00:49.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.960+0000 7f62c8f3d640 1 -- 172.21.15.71:0/2833172973 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c40ab420 con 0x7f62c40ae6f0 2024-03-29T11:00:49.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.960+0000 7f62c8f3d640 10 monclient: _finish_auth 0 2024-03-29T11:00:49.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.960+0000 7f62c8f3d640 10 monclient: _check_auth_tickets 2024-03-29T11:00:49.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.960+0000 7f62c8f3d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:49.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.960+0000 7f62cc1a3640 5 monclient: authenticate success, global_id 4245 2024-03-29T11:00:49.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.960+0000 7f62cc1a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:49.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.960+0000 7f62c8f3d640 1 -- 172.21.15.71:0/2833172973 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f62b4001690 con 0x7f62c40ae6f0 2024-03-29T11:00:49.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.960+0000 7f62c8f3d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:49.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.960+0000 7f62c8f3d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:00:49.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.960+0000 7f62c8f3d640 10 monclient: dump: 2024-03-29T11:00:49.948 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:00:49.948 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:49.948 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:49.948 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:49.948 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:49.948 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:49.948 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:49.948 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:49.949 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:49.949 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:00:49.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.960+0000 7f62cc1a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:49.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.960+0000 7f62c8f3d640 1 -- 172.21.15.71:0/2833172973 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62b4001920 con 0x7f62c40ae6f0 2024-03-29T11:00:49.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.960+0000 7f62c8f3d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:49.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.960+0000 7f62c8f3d640 1 -- 172.21.15.71:0/2833172973 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f62b4001cb0 con 0x7f62c40ae6f0 2024-03-29T11:00:49.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.960+0000 7f62c8f3d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:49.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.960+0000 7f62c8f3d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:49.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.960+0000 7f62c8f3d640 10 monclient: dump: 2024-03-29T11:00:49.949 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:00:49.949 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:49.950 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:49.950 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:49.950 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:49.950 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:49.950 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:49.950 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:49.950 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:49.950 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:00:49.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.960+0000 7f62cc1a3640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:49.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.960+0000 7f62cc1a3640 4 set_mon_vals no callback set 2024-03-29T11:00:49.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.961+0000 7f62cc1a3640 10 monclient: shutdown 2024-03-29T11:00:49.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.961+0000 7f62cc1a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:49.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.961+0000 7f62cc1a3640 1 -- 172.21.15.71:0/2833172973 >> v1:172.21.15.71:6789/0 conn(0x7f62c40ae6f0 legacy=0x7f62c414b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:49.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.961+0000 7f62cc1a3640 1 -- 172.21.15.71:0/2833172973 shutdown_connections 2024-03-29T11:00:49.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.961+0000 7f62cc1a3640 1 -- 172.21.15.71:0/2833172973 >> 172.21.15.71:0/2833172973 conn(0x7f62c40b35e0 msgr2=0x7f62c40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:49.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.961+0000 7f62cc1a3640 1 -- 172.21.15.71:0/2833172973 shutdown_connections 2024-03-29T11:00:49.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.961+0000 7f62cc1a3640 1 -- 172.21.15.71:0/2833172973 wait complete. 2024-03-29T11:00:49.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.962+0000 7f62cc1a3640 10 monclient: build_initial_monmap 2024-03-29T11:00:49.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.962+0000 7f62cc1a3640 10 monclient: monmap: 2024-03-29T11:00:49.951 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:00:49.951 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:49.951 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:00:49.951 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:00:49.951 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:49.951 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:49.951 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:49.951 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:00:49.951 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:00:49.951 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:00:49.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.962+0000 7f62cc1a3640 1 librados: starting msgr at 2024-03-29T11:00:49.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.962+0000 7f62cc1a3640 1 librados: starting objecter 2024-03-29T11:00:49.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.962+0000 7f62cc1a3640 1 Processor -- start 2024-03-29T11:00:49.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.962+0000 7f62cc1a3640 1 -- start start 2024-03-29T11:00:49.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.962+0000 7f62cc1a3640 1 librados: setting wanted keys 2024-03-29T11:00:49.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.962+0000 7f62cc1a3640 1 librados: calling monclient init 2024-03-29T11:00:49.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.962+0000 7f62cc1a3640 10 monclient: init 2024-03-29T11:00:49.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.963+0000 7f62cc1a3640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:49.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.963+0000 7f62cc1a3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:00:49.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.963+0000 7f62cc1a3640 10 monclient(hunting): picked mon.noname-c con 0x7f62c40a9930 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:49.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.963+0000 7f62cc1a3640 10 monclient(hunting): picked mon.noname-a con 0x7f62c40aa430 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:49.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.963+0000 7f62cc1a3640 10 monclient(hunting): picked mon.noname-b con 0x7f62c40ae6f0 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:49.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.963+0000 7f62cc1a3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c40acc20 con 0x7f62c40aa430 2024-03-29T11:00:49.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.963+0000 7f62cc1a3640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c41dbcb0 con 0x7f62c40ae6f0 2024-03-29T11:00:49.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.963+0000 7f62cc1a3640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c413a6c0 con 0x7f62c40a9930 2024-03-29T11:00:49.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.963+0000 7f62cc1a3640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:49.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.963+0000 7f62cc1a3640 10 monclient(hunting): authenticate will time out at 806.221576s 2024-03-29T11:00:49.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.964+0000 7f62c973e640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f62c40aa430 0x7f62c41db5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:59616/0 (socket says 172.21.15.71:59616) 2024-03-29T11:00:49.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.964+0000 7f62c973e640 1 -- 172.21.15.71:0/3787829037 learned_addr learned my addr 172.21.15.71:0/3787829037 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:00:49.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.964+0000 7f62ba7fc640 10 client.?.objecter ms_handle_connect 0x7f62c40aa430 2024-03-29T11:00:49.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.964+0000 7f62ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:49.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.965+0000 7f62ba7fc640 1 -- 172.21.15.71:0/3787829037 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434110947 0 0) 0x7f62c40acc20 con 0x7f62c40aa430 2024-03-29T11:00:49.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.965+0000 7f62ba7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:49.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.965+0000 7f62ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:49.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.965+0000 7f62ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:49.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.965+0000 7f62ba7fc640 10 monclient(hunting): my global_id is 4248 2024-03-29T11:00:49.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.965+0000 7f62ba7fc640 1 -- 172.21.15.71:0/3787829037 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62a8004eb0 con 0x7f62c40aa430 2024-03-29T11:00:49.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62ba7fc640 10 client.?.objecter ms_handle_connect 0x7f62c40ae6f0 2024-03-29T11:00:49.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:49.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62ba7fc640 1 -- 172.21.15.71:0/3787829037 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4143606229 0 0) 0x7f62a8004eb0 con 0x7f62c40aa430 2024-03-29T11:00:49.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62ba7fc640 1 -- 172.21.15.71:0/3787829037 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62c40acc20 con 0x7f62c40aa430 2024-03-29T11:00:49.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62ba7fc640 1 -- 172.21.15.71:0/3787829037 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f62bc001d10 con 0x7f62c40aa430 2024-03-29T11:00:49.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62ba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:49.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62ba7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:00:49.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62ba7fc640 10 monclient(hunting): dump: 2024-03-29T11:00:49.954 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:00:49.954 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:49.954 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:49.954 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:49.954 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:49.954 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:49.954 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:49.955 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:49.955 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:49.955 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:00:49.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62ba7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:49.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62ba7fc640 1 -- 172.21.15.71:0/3787829037 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2169214186 0 0) 0x7f62c41dbcb0 con 0x7f62c40ae6f0 2024-03-29T11:00:49.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62ba7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:49.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:49.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:49.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62ba7fc640 10 monclient(hunting): my global_id is 4237 2024-03-29T11:00:49.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62ba7fc640 1 -- 172.21.15.71:0/3787829037 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62a8004eb0 con 0x7f62c40ae6f0 2024-03-29T11:00:49.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62ba7fc640 1 -- 172.21.15.71:0/3787829037 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4056413196 0 0) 0x7f62c40acc20 con 0x7f62c40aa430 2024-03-29T11:00:49.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62ba7fc640 1 -- 172.21.15.71:0/3787829037 >> v1:172.21.15.71:6790/0 conn(0x7f62c40a9930 legacy=0x7f62c41dae90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:00:49.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62ba7fc640 1 -- 172.21.15.71:0/3787829037 >> v1:172.21.15.137:6789/0 conn(0x7f62c40ae6f0 legacy=0x7f62c41e2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:49.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62ba7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:00:49.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62ba7fc640 1 monclient: found mon.a 2024-03-29T11:00:49.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:49.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62ba7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:00:49.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62ba7fc640 1 -- 172.21.15.71:0/3787829037 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c41429b0 con 0x7f62c40aa430 2024-03-29T11:00:49.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62ba7fc640 10 monclient: _finish_auth 0 2024-03-29T11:00:49.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62ba7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:00:49.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:49.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62cc1a3640 5 monclient: authenticate success, global_id 4248 2024-03-29T11:00:49.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62cc1a3640 10 monclient: _renew_subs 2024-03-29T11:00:49.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62cc1a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:00:49.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62cc1a3640 1 -- 172.21.15.71:0/3787829037 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c41395c0 con 0x7f62c40aa430 2024-03-29T11:00:49.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62cc1a3640 10 client.4248.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:49.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62cc1a3640 10 monclient: _renew_subs 2024-03-29T11:00:49.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62cc1a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:00:49.957 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62cc1a3640 1 -- 172.21.15.71:0/3787829037 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c4137050 con 0x7f62c40aa430 2024-03-29T11:00:49.957 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.966+0000 7f62cc1a3640 1 librados: init done 2024-03-29T11:00:49.958 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.967+0000 7f62ba7fc640 1 -- 172.21.15.71:0/3787829037 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62bc001ff0 con 0x7f62c40aa430 2024-03-29T11:00:49.958 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.967+0000 7f62ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:49.958 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.967+0000 7f62bbfff640 4 set_mon_vals no callback set 2024-03-29T11:00:49.958 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.968+0000 7f62ba7fc640 1 -- 172.21.15.71:0/3787829037 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f62bc003de0 con 0x7f62c40aa430 2024-03-29T11:00:49.959 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.968+0000 7f62ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:49.959 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.968+0000 7f62ba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:49.959 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.968+0000 7f62ba7fc640 10 monclient: dump: 2024-03-29T11:00:49.959 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:00:49.959 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:49.959 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:49.959 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:49.959 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:49.959 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:49.959 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:49.959 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:49.959 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:49.959 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:00:49.959 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.968+0000 7f62cc1a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:49.959 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.968+0000 7f62cc1a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:49.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.968+0000 7f62cc1a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:00:49.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.968+0000 7f62cc1a3640 1 -- 172.21.15.71:0/3787829037 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62ac005080 con 0x7f62c40aa430 2024-03-29T11:00:49.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.970+0000 7f62ba7fc640 1 -- 172.21.15.71:0/3787829037 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f62bc003f40 con 0x7f62c40aa430 2024-03-29T11:00:49.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.970+0000 7f62ba7fc640 1 -- 172.21.15.71:0/3787829037 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f62bc084da0 con 0x7f62c40aa430 2024-03-29T11:00:49.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.970+0000 7f62ba7fc640 10 client.4248.objecter ms_dispatch 0x7f62c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:49.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.970+0000 7f62ba7fc640 3 client.4248.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:49.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.970+0000 7f62ba7fc640 3 client.4248.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:49.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.970+0000 7f62ba7fc640 20 client.4248.objecter dump_active .. 0 homeless 2024-03-29T11:00:49.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.970+0000 7f62c9f3f640 10 monclient: get_auth_request con 0x7f62a806dd00 auth_method 0 2024-03-29T11:00:49.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.971+0000 7f62ba7fc640 10 client.4248.objecter ms_handle_connect 0x7f62a806dd00 2024-03-29T11:00:49.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.976+0000 7f62ba7fc640 1 -- 172.21.15.71:0/3787829037 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f62bc050680 con 0x7f62c40aa430 2024-03-29T11:00:49.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.976+0000 7f62ba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:49.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:49.976+0000 7f62ba7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:50.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:50.285+0000 7f62cc1a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:50.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:50.285+0000 7f62cc1a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:50.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:50.285+0000 7f62cc1a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:00:50.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:50.285+0000 7f62cc1a3640 1 -- 172.21.15.71:0/3787829037 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62ac005330 con 0x7f62c40aa430 2024-03-29T11:00:50.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:50.286+0000 7f62ba7fc640 1 -- 172.21.15.71:0/3787829037 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f62bc0525b0 con 0x7f62c40aa430 2024-03-29T11:00:50.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:50.286+0000 7f62ba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:50.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:50.286+0000 7f62ba7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:50.266 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:00:50.266 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:50.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:50.289+0000 7f62cc1a3640 10 librados: watch_flush enter 2024-03-29T11:00:50.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:50.289+0000 7f62cc1a3640 10 librados: watch_flush exit 2024-03-29T11:00:50.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:50.289+0000 7f62cc1a3640 20 client.4248.objecter shutdown clearing up homeless session... 2024-03-29T11:00:50.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:50.289+0000 7f62cc1a3640 10 client.4248.objecter successfully canceled tick 2024-03-29T11:00:50.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:50.289+0000 7f62cc1a3640 1 -- 172.21.15.71:0/3787829037 >> v1:172.21.15.137:6816/973332665 conn(0x7f62a806dd00 legacy=0x7f62a80701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:50.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:50.289+0000 7f62cc1a3640 10 monclient: shutdown 2024-03-29T11:00:50.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:50.289+0000 7f62cc1a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:50.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:50.289+0000 7f62cc1a3640 1 -- 172.21.15.71:0/3787829037 >> v1:172.21.15.71:6789/0 conn(0x7f62c40aa430 legacy=0x7f62c41db5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:50.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:50.289+0000 7f62cc1a3640 1 -- 172.21.15.71:0/3787829037 shutdown_connections 2024-03-29T11:00:50.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:50.289+0000 7f62cc1a3640 1 -- 172.21.15.71:0/3787829037 >> 172.21.15.71:0/3787829037 conn(0x7f62c40b35e0 msgr2=0x7f62c40adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:50.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:50.289+0000 7f62cc1a3640 1 -- 172.21.15.71:0/3787829037 shutdown_connections 2024-03-29T11:00:50.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:50.289+0000 7f62cc1a3640 1 -- 172.21.15.71:0/3787829037 wait complete. 2024-03-29T11:00:50.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:50.289+0000 7f62cc1a3640 1 librados: shutdown 2024-03-29T11:00:50.283 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:57.089 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:57.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.258+0000 7f3a3dcaa640 10 monclient: get_monmap_and_config 2024-03-29T11:00:57.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.258+0000 7f3a3dcaa640 10 monclient: build_initial_monmap 2024-03-29T11:00:57.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.258+0000 7f3a3dcaa640 10 monclient: monmap: 2024-03-29T11:00:57.238 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:00:57.238 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:57.238 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:00:57.259129+0000 2024-03-29T11:00:57.238 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:00:57.259129+0000 2024-03-29T11:00:57.238 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:57.238 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:57.238 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:57.238 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:00:57.239 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:00:57.239 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:00:57.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.260+0000 7f3a3dcaa640 1 Processor -- start 2024-03-29T11:00:57.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.260+0000 7f3a3dcaa640 1 -- start start 2024-03-29T11:00:57.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.260+0000 7f3a3dcaa640 10 monclient: init 2024-03-29T11:00:57.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.261+0000 7f3a3dcaa640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:57.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.261+0000 7f3a3dcaa640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:00:57.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.261+0000 7f3a3dcaa640 10 monclient(hunting): picked mon.noname-a con 0x7f3a380b1f60 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:57.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.261+0000 7f3a3dcaa640 10 monclient(hunting): picked mon.noname-b con 0x7f3a38071eb0 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:57.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.261+0000 7f3a3dcaa640 10 monclient(hunting): picked mon.noname-c con 0x7f3a38143d10 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:57.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.261+0000 7f3a3dcaa640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a3806ef80 con 0x7f3a380b1f60 2024-03-29T11:00:57.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.261+0000 7f3a3dcaa640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a3806f260 con 0x7f3a38143d10 2024-03-29T11:00:57.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.261+0000 7f3a3dcaa640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a380b40c0 con 0x7f3a38071eb0 2024-03-29T11:00:57.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.261+0000 7f3a3dcaa640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:57.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.261+0000 7f3a3dcaa640 10 monclient(hunting): authenticate will time out at 813.519816s 2024-03-29T11:00:57.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.265+0000 7f3a377fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f3a380b1f60 0x7f3a38143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:36121/0 (socket says 172.21.15.71:36121) 2024-03-29T11:00:57.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.265+0000 7f3a377fe640 1 -- 172.21.15.71:0/199232034 learned_addr learned my addr 172.21.15.71:0/199232034 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:00:57.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.265+0000 7f3a367fc640 1 -- 172.21.15.71:0/199232034 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823174672 0 0) 0x7f3a380b40c0 con 0x7f3a38071eb0 2024-03-29T11:00:57.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.265+0000 7f3a367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:57.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.265+0000 7f3a367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:57.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.265+0000 7f3a367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:57.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.265+0000 7f3a367fc640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:00:57.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.265+0000 7f3a367fc640 1 -- 172.21.15.71:0/199232034 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a20003520 con 0x7f3a38071eb0 2024-03-29T11:00:57.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.265+0000 7f3a367fc640 1 -- 172.21.15.71:0/199232034 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009885407 0 0) 0x7f3a3806ef80 con 0x7f3a380b1f60 2024-03-29T11:00:57.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.265+0000 7f3a367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:57.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.265+0000 7f3a367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:57.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.265+0000 7f3a367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:57.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.265+0000 7f3a367fc640 10 monclient(hunting): my global_id is 4251 2024-03-29T11:00:57.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.265+0000 7f3a367fc640 1 -- 172.21.15.71:0/199232034 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a380b40c0 con 0x7f3a380b1f60 2024-03-29T11:00:57.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a367fc640 1 -- 172.21.15.71:0/199232034 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538319589 0 0) 0x7f3a3806f260 con 0x7f3a38143d10 2024-03-29T11:00:57.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:57.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:57.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:57.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a367fc640 10 monclient(hunting): my global_id is 4240 2024-03-29T11:00:57.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a367fc640 1 -- 172.21.15.71:0/199232034 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a3806ef80 con 0x7f3a38143d10 2024-03-29T11:00:57.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a367fc640 1 -- 172.21.15.71:0/199232034 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1107550947 0 0) 0x7f3a20003520 con 0x7f3a38071eb0 2024-03-29T11:00:57.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a367fc640 1 -- 172.21.15.71:0/199232034 >> v1:172.21.15.137:6789/0 conn(0x7f3a38143d10 legacy=0x7f3a38146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:57.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a367fc640 1 -- 172.21.15.71:0/199232034 >> v1:172.21.15.71:6789/0 conn(0x7f3a380b1f60 legacy=0x7f3a38143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:57.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a367fc640 10 monclient: _finish_hunting 0 2024-03-29T11:00:57.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a367fc640 1 monclient: found mon.noname-b 2024-03-29T11:00:57.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:57.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a367fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:00:57.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a367fc640 1 -- 172.21.15.71:0/199232034 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a38155d70 con 0x7f3a38071eb0 2024-03-29T11:00:57.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a367fc640 10 monclient: _finish_auth 0 2024-03-29T11:00:57.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a367fc640 10 monclient: _check_auth_tickets 2024-03-29T11:00:57.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:57.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a367fc640 1 -- 172.21.15.71:0/199232034 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f3a2c0015c0 con 0x7f3a38071eb0 2024-03-29T11:00:57.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:57.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a367fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:57.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a367fc640 10 monclient: dump: 2024-03-29T11:00:57.248 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:00:57.248 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:57.248 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:57.249 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:57.249 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:57.249 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:57.249 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:57.249 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:57.249 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:57.249 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:00:57.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a3dcaa640 5 monclient: authenticate success, global_id 4244 2024-03-29T11:00:57.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a3dcaa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:57.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a367fc640 1 -- 172.21.15.71:0/199232034 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a2c0017a0 con 0x7f3a38071eb0 2024-03-29T11:00:57.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:57.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a3dcaa640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:57.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a3dcaa640 4 set_mon_vals no callback set 2024-03-29T11:00:57.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a367fc640 1 -- 172.21.15.71:0/199232034 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f3a2c001bb0 con 0x7f3a38071eb0 2024-03-29T11:00:57.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:57.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:57.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.266+0000 7f3a367fc640 10 monclient: dump: 2024-03-29T11:00:57.250 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:00:57.250 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:57.250 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:57.250 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:57.250 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:57.250 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:57.250 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:57.250 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:57.250 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:57.250 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:00:57.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.267+0000 7f3a3dcaa640 10 monclient: shutdown 2024-03-29T11:00:57.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.267+0000 7f3a3dcaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:57.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.267+0000 7f3a3dcaa640 1 -- 172.21.15.71:0/199232034 >> v1:172.21.15.71:6790/0 conn(0x7f3a38071eb0 legacy=0x7f3a38146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:57.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.267+0000 7f3a3dcaa640 1 -- 172.21.15.71:0/199232034 shutdown_connections 2024-03-29T11:00:57.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.267+0000 7f3a3dcaa640 1 -- 172.21.15.71:0/199232034 >> 172.21.15.71:0/199232034 conn(0x7f3a38066a00 msgr2=0x7f3a38066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:57.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.267+0000 7f3a3dcaa640 1 -- 172.21.15.71:0/199232034 shutdown_connections 2024-03-29T11:00:57.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.267+0000 7f3a3dcaa640 1 -- 172.21.15.71:0/199232034 wait complete. 2024-03-29T11:00:57.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.268+0000 7f3a3dcaa640 10 monclient: build_initial_monmap 2024-03-29T11:00:57.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.268+0000 7f3a3dcaa640 10 monclient: monmap: 2024-03-29T11:00:57.252 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:00:57.253 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:57.253 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:00:57.253 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:00:57.253 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:57.253 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:57.253 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:00:57.253 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:00:57.253 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:00:57.253 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:00:57.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.269+0000 7f3a3dcaa640 1 librados: starting msgr at 2024-03-29T11:00:57.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.269+0000 7f3a3dcaa640 1 librados: starting objecter 2024-03-29T11:00:57.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.269+0000 7f3a3dcaa640 1 Processor -- start 2024-03-29T11:00:57.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.269+0000 7f3a3dcaa640 1 -- start start 2024-03-29T11:00:57.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.269+0000 7f3a3dcaa640 1 librados: setting wanted keys 2024-03-29T11:00:57.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.269+0000 7f3a3dcaa640 1 librados: calling monclient init 2024-03-29T11:00:57.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.269+0000 7f3a3dcaa640 10 monclient: init 2024-03-29T11:00:57.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.270+0000 7f3a3dcaa640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:57.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.270+0000 7f3a3dcaa640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:00:57.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.270+0000 7f3a3dcaa640 10 monclient(hunting): picked mon.noname-b con 0x7f3a380b1f60 addr v1:172.21.15.137:6789/0 2024-03-29T11:00:57.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.270+0000 7f3a3dcaa640 10 monclient(hunting): picked mon.noname-c con 0x7f3a38143d10 addr v1:172.21.15.71:6790/0 2024-03-29T11:00:57.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.270+0000 7f3a3dcaa640 10 monclient(hunting): picked mon.noname-a con 0x7f3a3813cbe0 addr v1:172.21.15.71:6789/0 2024-03-29T11:00:57.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.270+0000 7f3a3dcaa640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a38136950 con 0x7f3a3813cbe0 2024-03-29T11:00:57.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.270+0000 7f3a3dcaa640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a380ad3a0 con 0x7f3a380b1f60 2024-03-29T11:00:57.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.270+0000 7f3a3dcaa640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a380abc10 con 0x7f3a38143d10 2024-03-29T11:00:57.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.270+0000 7f3a3dcaa640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:57.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.270+0000 7f3a3dcaa640 10 monclient(hunting): authenticate will time out at 813.528698s 2024-03-29T11:00:57.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.270+0000 7f3a377fe640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f3a380b1f60 0x7f3a381547e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:55990/0 (socket says 172.21.15.71:55990) 2024-03-29T11:00:57.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.270+0000 7f3a377fe640 1 -- 172.21.15.71:0/2376652987 learned_addr learned my addr 172.21.15.71:0/2376652987 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:00:57.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.271+0000 7f3a1ffff640 10 client.?.objecter ms_handle_connect 0x7f3a380b1f60 2024-03-29T11:00:57.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.271+0000 7f3a1ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:57.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.271+0000 7f3a1ffff640 1 -- 172.21.15.71:0/2376652987 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1759178348 0 0) 0x7f3a380ad3a0 con 0x7f3a380b1f60 2024-03-29T11:00:57.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.271+0000 7f3a1ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:57.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.271+0000 7f3a1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:57.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.271+0000 7f3a1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:57.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.271+0000 7f3a1ffff640 10 monclient(hunting): my global_id is 4243 2024-03-29T11:00:57.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.271+0000 7f3a1ffff640 1 -- 172.21.15.71:0/2376652987 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a24004eb0 con 0x7f3a380b1f60 2024-03-29T11:00:57.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.272+0000 7f3a1ffff640 1 -- 172.21.15.71:0/2376652987 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2341863617 0 0) 0x7f3a24004eb0 con 0x7f3a380b1f60 2024-03-29T11:00:57.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.272+0000 7f3a1ffff640 1 -- 172.21.15.71:0/2376652987 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a380ad3a0 con 0x7f3a380b1f60 2024-03-29T11:00:57.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.272+0000 7f3a1ffff640 1 -- 172.21.15.71:0/2376652987 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f3a30001cf0 con 0x7f3a380b1f60 2024-03-29T11:00:57.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.272+0000 7f3a1ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:57.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.272+0000 7f3a1ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:57.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.272+0000 7f3a1ffff640 10 monclient(hunting): dump: 2024-03-29T11:00:57.255 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:00:57.256 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:57.256 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:57.256 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:57.256 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:57.256 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:57.256 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:57.256 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:57.256 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:57.256 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:00:57.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.272+0000 7f3a1ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:57.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.273+0000 7f3a1ffff640 1 -- 172.21.15.71:0/2376652987 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3873039949 0 0) 0x7f3a380ad3a0 con 0x7f3a380b1f60 2024-03-29T11:00:57.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.273+0000 7f3a1ffff640 1 -- 172.21.15.71:0/2376652987 >> v1:172.21.15.71:6790/0 conn(0x7f3a38143d10 legacy=0x7f3a3813c4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:00:57.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.275+0000 7f3a1ffff640 1 -- 172.21.15.71:0/2376652987 >> v1:172.21.15.71:6789/0 conn(0x7f3a3813cbe0 legacy=0x7f3a381ddc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:57.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.275+0000 7f3a1ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:00:57.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.275+0000 7f3a1ffff640 1 monclient: found mon.b 2024-03-29T11:00:57.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.275+0000 7f3a1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:57.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.275+0000 7f3a1ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:57.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.275+0000 7f3a1ffff640 1 -- 172.21.15.71:0/2376652987 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a380aacd0 con 0x7f3a380b1f60 2024-03-29T11:00:57.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.275+0000 7f3a1ffff640 10 monclient: _finish_auth 0 2024-03-29T11:00:57.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.275+0000 7f3a1ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:00:57.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.275+0000 7f3a1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:57.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.275+0000 7f3a1ffff640 10 client.?.objecter ms_handle_connect 0x7f3a3813cbe0 2024-03-29T11:00:57.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.275+0000 7f3a1ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:57.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.275+0000 7f3a3dcaa640 5 monclient: authenticate success, global_id 4243 2024-03-29T11:00:57.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.275+0000 7f3a3dcaa640 10 monclient: _renew_subs 2024-03-29T11:00:57.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.275+0000 7f3a3dcaa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:57.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.275+0000 7f3a3dcaa640 1 -- 172.21.15.71:0/2376652987 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a380ac130 con 0x7f3a380b1f60 2024-03-29T11:00:57.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.275+0000 7f3a3dcaa640 10 client.4243.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:57.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.275+0000 7f3a3dcaa640 10 monclient: _renew_subs 2024-03-29T11:00:57.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.275+0000 7f3a3dcaa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:57.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.275+0000 7f3a3dcaa640 1 -- 172.21.15.71:0/2376652987 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a380ab670 con 0x7f3a380b1f60 2024-03-29T11:00:57.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.275+0000 7f3a3dcaa640 1 librados: init done 2024-03-29T11:00:57.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.276+0000 7f3a1ffff640 1 -- 172.21.15.71:0/2376652987 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a30002050 con 0x7f3a380b1f60 2024-03-29T11:00:57.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.276+0000 7f3a1ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:57.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.276+0000 7f3a1ffff640 1 -- 172.21.15.71:0/2376652987 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f3a30004fb0 con 0x7f3a380b1f60 2024-03-29T11:00:57.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.276+0000 7f3a1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:57.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.276+0000 7f3a1ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:57.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.276+0000 7f3a1ffff640 10 monclient: dump: 2024-03-29T11:00:57.260 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:00:57.260 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:00:57.260 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:57.261 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:00:57.261 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:57.261 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:00:57.261 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:00:57.261 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:00:57.261 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:00:57.261 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:00:57.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.277+0000 7f3a357fa640 4 set_mon_vals no callback set 2024-03-29T11:00:57.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.278+0000 7f3a1ffff640 1 -- 172.21.15.71:0/2376652987 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f3a3001af80 con 0x7f3a380b1f60 2024-03-29T11:00:57.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.278+0000 7f3a3dcaa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:57.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.278+0000 7f3a3dcaa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:57.262 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.279+0000 7f3a1ffff640 1 -- 172.21.15.71:0/2376652987 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f3a30084ef0 con 0x7f3a380b1f60 2024-03-29T11:00:57.262 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.279+0000 7f3a1ffff640 10 client.4243.objecter ms_dispatch 0x7f3a38000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:57.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.279+0000 7f3a1ffff640 3 client.4243.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:57.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.279+0000 7f3a1ffff640 3 client.4243.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:57.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.279+0000 7f3a1ffff640 20 client.4243.objecter dump_active .. 0 homeless 2024-03-29T11:00:57.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.279+0000 7f3a3dcaa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:57.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.279+0000 7f3a3dcaa640 1 -- 172.21.15.71:0/2376652987 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a18005080 con 0x7f3a380b1f60 2024-03-29T11:00:57.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.280+0000 7f3a36ffd640 10 monclient: get_auth_request con 0x7f3a2406dc00 auth_method 0 2024-03-29T11:00:57.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.281+0000 7f3a1ffff640 10 client.4243.objecter ms_handle_connect 0x7f3a2406dc00 2024-03-29T11:00:57.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.288+0000 7f3a1ffff640 1 -- 172.21.15.71:0/2376652987 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3a30050830 con 0x7f3a380b1f60 2024-03-29T11:00:57.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.288+0000 7f3a1ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:57.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.288+0000 7f3a1ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:57.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.624+0000 7f3a3dcaa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:57.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.624+0000 7f3a3dcaa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:57.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.624+0000 7f3a3dcaa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:00:57.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.624+0000 7f3a3dcaa640 1 -- 172.21.15.71:0/2376652987 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a18005330 con 0x7f3a380b1f60 2024-03-29T11:00:57.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.626+0000 7f3a1ffff640 1 -- 172.21.15.71:0/2376652987 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f3a300845a0 con 0x7f3a380b1f60 2024-03-29T11:00:57.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.626+0000 7f3a1ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:57.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.626+0000 7f3a1ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:57.606 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:00:57.606 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:57.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.629+0000 7f3a3dcaa640 10 librados: watch_flush enter 2024-03-29T11:00:57.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.629+0000 7f3a3dcaa640 10 librados: watch_flush exit 2024-03-29T11:00:57.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.629+0000 7f3a3dcaa640 20 client.4243.objecter shutdown clearing up homeless session... 2024-03-29T11:00:57.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.629+0000 7f3a3dcaa640 10 client.4243.objecter successfully canceled tick 2024-03-29T11:00:57.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.630+0000 7f3a3dcaa640 1 -- 172.21.15.71:0/2376652987 >> v1:172.21.15.137:6816/973332665 conn(0x7f3a2406dc00 legacy=0x7f3a240700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:57.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.630+0000 7f3a3dcaa640 10 monclient: shutdown 2024-03-29T11:00:57.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.630+0000 7f3a3dcaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:57.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.630+0000 7f3a3dcaa640 1 -- 172.21.15.71:0/2376652987 >> v1:172.21.15.137:6789/0 conn(0x7f3a380b1f60 legacy=0x7f3a381547e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:57.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.630+0000 7f3a3dcaa640 1 -- 172.21.15.71:0/2376652987 shutdown_connections 2024-03-29T11:00:57.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.630+0000 7f3a3dcaa640 1 -- 172.21.15.71:0/2376652987 >> 172.21.15.71:0/2376652987 conn(0x7f3a38066a00 msgr2=0x7f3a380b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:57.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.631+0000 7f3a3dcaa640 1 -- 172.21.15.71:0/2376652987 shutdown_connections 2024-03-29T11:00:57.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.631+0000 7f3a3dcaa640 1 -- 172.21.15.71:0/2376652987 wait complete. 2024-03-29T11:00:57.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:00:57.631+0000 7f3a3dcaa640 1 librados: shutdown 2024-03-29T11:00:57.628 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:04.434 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:04.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.608+0000 7f4e2a1b9640 10 monclient: get_monmap_and_config 2024-03-29T11:01:04.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.608+0000 7f4e2a1b9640 10 monclient: build_initial_monmap 2024-03-29T11:01:04.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.608+0000 7f4e2a1b9640 10 monclient: monmap: 2024-03-29T11:01:04.589 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:01:04.590 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:04.590 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:01:04.609480+0000 2024-03-29T11:01:04.590 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:01:04.609480+0000 2024-03-29T11:01:04.590 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:04.590 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:04.590 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:01:04.590 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:01:04.590 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:01:04.590 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:04.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.610+0000 7f4e2a1b9640 1 Processor -- start 2024-03-29T11:01:04.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.610+0000 7f4e2a1b9640 1 -- start start 2024-03-29T11:01:04.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.610+0000 7f4e2a1b9640 10 monclient: init 2024-03-29T11:01:04.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.611+0000 7f4e2a1b9640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:04.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.611+0000 7f4e2a1b9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:01:04.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.611+0000 7f4e2a1b9640 10 monclient(hunting): picked mon.noname-b con 0x7f4e240ae250 addr v1:172.21.15.71:6790/0 2024-03-29T11:01:04.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.611+0000 7f4e2a1b9640 10 monclient(hunting): picked mon.noname-a con 0x7f4e2413f970 addr v1:172.21.15.71:6789/0 2024-03-29T11:01:04.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.611+0000 7f4e2a1b9640 10 monclient(hunting): picked mon.noname-c con 0x7f4e24142800 addr v1:172.21.15.137:6789/0 2024-03-29T11:01:04.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.611+0000 7f4e2a1b9640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e240afe00 con 0x7f4e2413f970 2024-03-29T11:01:04.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.611+0000 7f4e2a1b9640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e24142c10 con 0x7f4e24142800 2024-03-29T11:01:04.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.611+0000 7f4e2a1b9640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e240afb20 con 0x7f4e240ae250 2024-03-29T11:01:04.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.611+0000 7f4e2a1b9640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:04.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.611+0000 7f4e2a1b9640 10 monclient(hunting): authenticate will time out at 820.869567s 2024-03-29T11:01:04.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.616+0000 7f4e237fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f4e240ae250 0x7f4e2413f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:34461/0 (socket says 172.21.15.71:34461) 2024-03-29T11:01:04.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.616+0000 7f4e237fe640 1 -- 172.21.15.71:0/3160257744 learned_addr learned my addr 172.21.15.71:0/3160257744 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:01:04.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.618+0000 7f4e227fc640 1 -- 172.21.15.71:0/3160257744 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 136994150 0 0) 0x7f4e240afb20 con 0x7f4e240ae250 2024-03-29T11:01:04.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.618+0000 7f4e227fc640 10 monclient(hunting): _negotiate 2024-03-29T11:01:04.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.618+0000 7f4e227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:04.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.618+0000 7f4e227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:04.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.618+0000 7f4e227fc640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:01:04.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.618+0000 7f4e227fc640 1 -- 172.21.15.71:0/3160257744 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e0c003520 con 0x7f4e240ae250 2024-03-29T11:01:04.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.619+0000 7f4e227fc640 1 -- 172.21.15.71:0/3160257744 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 97456389 0 0) 0x7f4e0c003520 con 0x7f4e240ae250 2024-03-29T11:01:04.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.619+0000 7f4e227fc640 1 -- 172.21.15.71:0/3160257744 >> v1:172.21.15.137:6789/0 conn(0x7f4e24142800 legacy=0x7f4e2414da10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:01:04.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.619+0000 7f4e227fc640 1 -- 172.21.15.71:0/3160257744 >> v1:172.21.15.71:6789/0 conn(0x7f4e2413f970 legacy=0x7f4e241420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:04.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.619+0000 7f4e227fc640 10 monclient: _finish_hunting 0 2024-03-29T11:01:04.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.619+0000 7f4e227fc640 1 monclient: found mon.noname-b 2024-03-29T11:01:04.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.619+0000 7f4e227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:04.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.619+0000 7f4e227fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:01:04.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.619+0000 7f4e227fc640 1 -- 172.21.15.71:0/3160257744 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e240aab80 con 0x7f4e240ae250 2024-03-29T11:01:04.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.619+0000 7f4e227fc640 10 monclient: _finish_auth 0 2024-03-29T11:01:04.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.619+0000 7f4e227fc640 10 monclient: _check_auth_tickets 2024-03-29T11:01:04.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.619+0000 7f4e227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:04.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.619+0000 7f4e227fc640 1 -- 172.21.15.71:0/3160257744 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4e1c001a10 con 0x7f4e240ae250 2024-03-29T11:01:04.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.619+0000 7f4e227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:04.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.619+0000 7f4e227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:04.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.619+0000 7f4e227fc640 10 monclient: dump: 2024-03-29T11:01:04.600 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:04.600 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:04.600 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:04.600 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:04.600 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:04.600 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:04.600 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:04.600 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:04.600 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:04.601 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:04.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.619+0000 7f4e227fc640 1 -- 172.21.15.71:0/3160257744 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e1c001ca0 con 0x7f4e240ae250 2024-03-29T11:01:04.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.619+0000 7f4e227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:04.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.619+0000 7f4e227fc640 1 -- 172.21.15.71:0/3160257744 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4e1c002030 con 0x7f4e240ae250 2024-03-29T11:01:04.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.619+0000 7f4e227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:04.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.619+0000 7f4e227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:04.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.619+0000 7f4e227fc640 10 monclient: dump: 2024-03-29T11:01:04.601 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:04.601 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:04.601 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:04.601 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:04.601 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:04.602 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:04.602 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:04.602 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:04.602 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:04.602 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:04.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.621+0000 7f4e2a1b9640 5 monclient: authenticate success, global_id 4247 2024-03-29T11:01:04.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.621+0000 7f4e2a1b9640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:04.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.621+0000 7f4e2a1b9640 4 set_mon_vals no callback set 2024-03-29T11:01:04.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.622+0000 7f4e2a1b9640 10 monclient: shutdown 2024-03-29T11:01:04.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.622+0000 7f4e2a1b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:04.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.622+0000 7f4e2a1b9640 1 -- 172.21.15.71:0/3160257744 >> v1:172.21.15.71:6790/0 conn(0x7f4e240ae250 legacy=0x7f4e2413f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:04.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.622+0000 7f4e2a1b9640 1 -- 172.21.15.71:0/3160257744 shutdown_connections 2024-03-29T11:01:04.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.622+0000 7f4e2a1b9640 1 -- 172.21.15.71:0/3160257744 >> 172.21.15.71:0/3160257744 conn(0x7f4e240b35e0 msgr2=0x7f4e240b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:04.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.622+0000 7f4e2a1b9640 1 -- 172.21.15.71:0/3160257744 shutdown_connections 2024-03-29T11:01:04.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.623+0000 7f4e2a1b9640 1 -- 172.21.15.71:0/3160257744 wait complete. 2024-03-29T11:01:04.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.623+0000 7f4e2a1b9640 10 monclient: build_initial_monmap 2024-03-29T11:01:04.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.623+0000 7f4e2a1b9640 10 monclient: monmap: 2024-03-29T11:01:04.605 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:01:04.605 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:04.605 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:01:04.605 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:01:04.605 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:04.605 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:04.605 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:01:04.605 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:01:04.605 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:01:04.605 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:04.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.624+0000 7f4e2a1b9640 1 librados: starting msgr at 2024-03-29T11:01:04.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.624+0000 7f4e2a1b9640 1 librados: starting objecter 2024-03-29T11:01:04.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.624+0000 7f4e2a1b9640 1 Processor -- start 2024-03-29T11:01:04.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.624+0000 7f4e2a1b9640 1 -- start start 2024-03-29T11:01:04.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.624+0000 7f4e2a1b9640 1 librados: setting wanted keys 2024-03-29T11:01:04.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.624+0000 7f4e2a1b9640 1 librados: calling monclient init 2024-03-29T11:01:04.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.624+0000 7f4e2a1b9640 10 monclient: init 2024-03-29T11:01:04.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.625+0000 7f4e2a1b9640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:04.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.625+0000 7f4e2a1b9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:01:04.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.627+0000 7f4e2a1b9640 10 monclient(hunting): picked mon.noname-b con 0x7f4e240ae250 addr v1:172.21.15.137:6789/0 2024-03-29T11:01:04.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.627+0000 7f4e2a1b9640 10 monclient(hunting): picked mon.noname-a con 0x7f4e2413f970 addr v1:172.21.15.71:6789/0 2024-03-29T11:01:04.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.627+0000 7f4e2a1b9640 10 monclient(hunting): picked mon.noname-c con 0x7f4e24142800 addr v1:172.21.15.71:6790/0 2024-03-29T11:01:04.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.627+0000 7f4e2a1b9640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e241dbd50 con 0x7f4e2413f970 2024-03-29T11:01:04.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.627+0000 7f4e2a1b9640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e24146010 con 0x7f4e240ae250 2024-03-29T11:01:04.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.627+0000 7f4e2a1b9640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e241462f0 con 0x7f4e24142800 2024-03-29T11:01:04.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.627+0000 7f4e2a1b9640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:04.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.627+0000 7f4e2a1b9640 10 monclient(hunting): authenticate will time out at 820.885451s 2024-03-29T11:01:04.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.628+0000 7f4e237fe640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f4e240ae250 0x7f4e24144170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:49413/0 (socket says 172.21.15.71:49413) 2024-03-29T11:01:04.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.628+0000 7f4e237fe640 1 -- 172.21.15.71:0/2258988824 learned_addr learned my addr 172.21.15.71:0/2258988824 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:01:04.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.628+0000 7f4e0bfff640 10 client.?.objecter ms_handle_connect 0x7f4e240ae250 2024-03-29T11:01:04.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.628+0000 7f4e0bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:04.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.628+0000 7f4e0bfff640 1 -- 172.21.15.71:0/2258988824 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743126394 0 0) 0x7f4e24146010 con 0x7f4e240ae250 2024-03-29T11:01:04.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.628+0000 7f4e0bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:04.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.628+0000 7f4e0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:04.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.628+0000 7f4e0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:04.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.628+0000 7f4e0bfff640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:01:04.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.628+0000 7f4e0bfff640 1 -- 172.21.15.71:0/2258988824 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e10004eb0 con 0x7f4e240ae250 2024-03-29T11:01:04.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.629+0000 7f4e0bfff640 1 -- 172.21.15.71:0/2258988824 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 941638461 0 0) 0x7f4e10004eb0 con 0x7f4e240ae250 2024-03-29T11:01:04.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.630+0000 7f4e0bfff640 1 -- 172.21.15.71:0/2258988824 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e24146010 con 0x7f4e240ae250 2024-03-29T11:01:04.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.630+0000 7f4e0bfff640 1 -- 172.21.15.71:0/2258988824 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4e1c0018d0 con 0x7f4e240ae250 2024-03-29T11:01:04.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.630+0000 7f4e0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:04.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.630+0000 7f4e0bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:04.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.630+0000 7f4e0bfff640 10 monclient(hunting): dump: 2024-03-29T11:01:04.616 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:04.616 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:04.616 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:04.616 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:04.616 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:04.617 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:04.617 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:04.617 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:04.617 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:04.617 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:04.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.630+0000 7f4e0bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:04.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.630+0000 7f4e0bfff640 1 -- 172.21.15.71:0/2258988824 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 328701881 0 0) 0x7f4e24146010 con 0x7f4e240ae250 2024-03-29T11:01:04.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.630+0000 7f4e0bfff640 1 -- 172.21.15.71:0/2258988824 >> v1:172.21.15.71:6790/0 conn(0x7f4e24142800 legacy=0x7f4e241e5e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:01:04.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.631+0000 7f4e0bfff640 1 -- 172.21.15.71:0/2258988824 >> v1:172.21.15.71:6789/0 conn(0x7f4e2413f970 legacy=0x7f4e241db640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:04.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.631+0000 7f4e0bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:01:04.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.631+0000 7f4e0bfff640 1 monclient: found mon.b 2024-03-29T11:01:04.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.631+0000 7f4e0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:04.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.631+0000 7f4e0bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:01:04.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.631+0000 7f4e0bfff640 1 -- 172.21.15.71:0/2258988824 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e24144bf0 con 0x7f4e240ae250 2024-03-29T11:01:04.618 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.631+0000 7f4e0bfff640 10 monclient: _finish_auth 0 2024-03-29T11:01:04.618 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.631+0000 7f4e0bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:01:04.618 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.631+0000 7f4e0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:04.618 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.631+0000 7f4e0bfff640 10 client.?.objecter ms_handle_connect 0x7f4e2413f970 2024-03-29T11:01:04.618 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.631+0000 7f4e0bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:04.618 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.631+0000 7f4e2a1b9640 5 monclient: authenticate success, global_id 4246 2024-03-29T11:01:04.618 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.631+0000 7f4e2a1b9640 10 monclient: _renew_subs 2024-03-29T11:01:04.618 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.631+0000 7f4e2a1b9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:01:04.618 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.631+0000 7f4e0bfff640 1 -- 172.21.15.71:0/2258988824 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e1c001a30 con 0x7f4e240ae250 2024-03-29T11:01:04.618 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.631+0000 7f4e2a1b9640 1 -- 172.21.15.71:0/2258988824 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e2411f200 con 0x7f4e240ae250 2024-03-29T11:01:04.618 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.632+0000 7f4e0bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:04.618 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.632+0000 7f4e0bfff640 1 -- 172.21.15.71:0/2258988824 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4e1c0051d0 con 0x7f4e240ae250 2024-03-29T11:01:04.618 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.632+0000 7f4e0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:04.618 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.632+0000 7f4e0bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:04.618 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.632+0000 7f4e0bfff640 10 monclient: dump: 2024-03-29T11:01:04.619 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:04.619 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:04.619 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:04.619 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:04.619 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:04.619 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:04.619 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:04.619 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:04.619 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:04.619 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:04.619 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.632+0000 7f4e2a1b9640 10 client.4246.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:04.619 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.632+0000 7f4e2a1b9640 10 monclient: _renew_subs 2024-03-29T11:01:04.619 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.632+0000 7f4e2a1b9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:01:04.619 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.632+0000 7f4e2a1b9640 1 -- 172.21.15.71:0/2258988824 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e241452f0 con 0x7f4e240ae250 2024-03-29T11:01:04.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.632+0000 7f4e2a1b9640 1 librados: init done 2024-03-29T11:01:04.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.634+0000 7f4e0bfff640 1 -- 172.21.15.71:0/2258988824 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f4e1c01b1a0 con 0x7f4e240ae250 2024-03-29T11:01:04.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.634+0000 7f4e217fa640 4 set_mon_vals no callback set 2024-03-29T11:01:04.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.635+0000 7f4e22ffd640 10 monclient: get_auth_request con 0x7f4e1006dd30 auth_method 0 2024-03-29T11:01:04.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.635+0000 7f4e0bfff640 1 -- 172.21.15.71:0/2258988824 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f4e1c084fb0 con 0x7f4e240ae250 2024-03-29T11:01:04.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.635+0000 7f4e0bfff640 10 client.4246.objecter ms_dispatch 0x7f4e24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:04.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.635+0000 7f4e0bfff640 3 client.4246.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:04.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.635+0000 7f4e0bfff640 3 client.4246.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:04.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.635+0000 7f4e0bfff640 20 client.4246.objecter dump_active .. 0 homeless 2024-03-29T11:01:04.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.635+0000 7f4e0bfff640 10 client.4246.objecter ms_handle_connect 0x7f4e1006dd30 2024-03-29T11:01:04.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.636+0000 7f4e2a1b9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:04.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.636+0000 7f4e2a1b9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:04.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.636+0000 7f4e2a1b9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:01:04.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.636+0000 7f4e2a1b9640 1 -- 172.21.15.71:0/2258988824 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e04005080 con 0x7f4e240ae250 2024-03-29T11:01:04.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.646+0000 7f4e0bfff640 1 -- 172.21.15.71:0/2258988824 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4e1c050890 con 0x7f4e240ae250 2024-03-29T11:01:04.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.646+0000 7f4e0bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:04.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.646+0000 7f4e0bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:04.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.975+0000 7f4e2a1b9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:04.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.976+0000 7f4e2a1b9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:04.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.976+0000 7f4e2a1b9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:01:04.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.976+0000 7f4e2a1b9640 1 -- 172.21.15.71:0/2258988824 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e04005330 con 0x7f4e240ae250 2024-03-29T11:01:04.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.977+0000 7f4e0bfff640 1 -- 172.21.15.71:0/2258988824 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f4e1c0527c0 con 0x7f4e240ae250 2024-03-29T11:01:04.957 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.977+0000 7f4e0bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:04.957 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.977+0000 7f4e0bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:04.957 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:01:04.957 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:04.963 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.980+0000 7f4e2a1b9640 10 librados: watch_flush enter 2024-03-29T11:01:04.963 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.980+0000 7f4e2a1b9640 10 librados: watch_flush exit 2024-03-29T11:01:04.963 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.980+0000 7f4e2a1b9640 20 client.4246.objecter shutdown clearing up homeless session... 2024-03-29T11:01:04.963 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.980+0000 7f4e2a1b9640 10 client.4246.objecter successfully canceled tick 2024-03-29T11:01:04.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.984+0000 7f4e2a1b9640 1 -- 172.21.15.71:0/2258988824 >> v1:172.21.15.137:6816/973332665 conn(0x7f4e1006dd30 legacy=0x7f4e100701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:04.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.984+0000 7f4e2a1b9640 10 monclient: shutdown 2024-03-29T11:01:04.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.984+0000 7f4e2a1b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:04.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.984+0000 7f4e2a1b9640 1 -- 172.21.15.71:0/2258988824 >> v1:172.21.15.137:6789/0 conn(0x7f4e240ae250 legacy=0x7f4e24144170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:04.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.984+0000 7f4e2a1b9640 1 -- 172.21.15.71:0/2258988824 shutdown_connections 2024-03-29T11:01:04.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.984+0000 7f4e2a1b9640 1 -- 172.21.15.71:0/2258988824 >> 172.21.15.71:0/2258988824 conn(0x7f4e240b35e0 msgr2=0x7f4e24142000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:04.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.984+0000 7f4e2a1b9640 1 -- 172.21.15.71:0/2258988824 shutdown_connections 2024-03-29T11:01:04.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.984+0000 7f4e2a1b9640 1 -- 172.21.15.71:0/2258988824 wait complete. 2024-03-29T11:01:04.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:04.984+0000 7f4e2a1b9640 1 librados: shutdown 2024-03-29T11:01:04.980 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:11.786 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:11.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.964+0000 7f9b656e4640 10 monclient: get_monmap_and_config 2024-03-29T11:01:11.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.964+0000 7f9b656e4640 10 monclient: build_initial_monmap 2024-03-29T11:01:11.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.964+0000 7f9b656e4640 10 monclient: monmap: 2024-03-29T11:01:11.945 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:01:11.945 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:11.945 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:01:11.965732+0000 2024-03-29T11:01:11.945 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:01:11.965732+0000 2024-03-29T11:01:11.945 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:11.945 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:11.945 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:01:11.945 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:01:11.945 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:01:11.945 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:11.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.967+0000 7f9b656e4640 1 Processor -- start 2024-03-29T11:01:11.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.968+0000 7f9b656e4640 1 -- start start 2024-03-29T11:01:11.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.968+0000 7f9b656e4640 10 monclient: init 2024-03-29T11:01:11.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.969+0000 7f9b656e4640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:11.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.969+0000 7f9b656e4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:01:11.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.969+0000 7f9b656e4640 10 monclient(hunting): picked mon.noname-b con 0x7f9b58038a00 addr v1:172.21.15.71:6790/0 2024-03-29T11:01:11.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.969+0000 7f9b656e4640 10 monclient(hunting): picked mon.noname-c con 0x7f9b58038de0 addr v1:172.21.15.137:6789/0 2024-03-29T11:01:11.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.969+0000 7f9b656e4640 10 monclient(hunting): picked mon.noname-a con 0x7f9b58030680 addr v1:172.21.15.71:6789/0 2024-03-29T11:01:11.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.969+0000 7f9b656e4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b580c54e0 con 0x7f9b58030680 2024-03-29T11:01:11.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.969+0000 7f9b656e4640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b58036450 con 0x7f9b58038de0 2024-03-29T11:01:11.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.969+0000 7f9b656e4640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b58039860 con 0x7f9b58038a00 2024-03-29T11:01:11.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.969+0000 7f9b656e4640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:11.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.969+0000 7f9b656e4640 10 monclient(hunting): authenticate will time out at 828.227975s 2024-03-29T11:01:11.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.972+0000 7f9b5f7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f9b58030680 0x7f9b580d3190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:50369/0 (socket says 172.21.15.71:50369) 2024-03-29T11:01:11.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.972+0000 7f9b5f7fe640 1 -- 172.21.15.71:0/637627832 learned_addr learned my addr 172.21.15.71:0/637627832 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:01:11.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.973+0000 7f9b5dffb640 1 -- 172.21.15.71:0/637627832 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442190459 0 0) 0x7f9b580c54e0 con 0x7f9b58030680 2024-03-29T11:01:11.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.973+0000 7f9b5dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:11.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.973+0000 7f9b5dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:11.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.973+0000 7f9b5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:11.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.973+0000 7f9b5dffb640 10 monclient(hunting): my global_id is 4260 2024-03-29T11:01:11.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.973+0000 7f9b5dffb640 1 -- 172.21.15.71:0/637627832 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b48003520 con 0x7f9b58030680 2024-03-29T11:01:11.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.973+0000 7f9b5dffb640 1 -- 172.21.15.71:0/637627832 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 122213347 0 0) 0x7f9b48003520 con 0x7f9b58030680 2024-03-29T11:01:11.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.973+0000 7f9b5dffb640 1 -- 172.21.15.71:0/637627832 >> v1:172.21.15.71:6790/0 conn(0x7f9b58038a00 legacy=0x7f9b58033540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:11.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.973+0000 7f9b5dffb640 1 -- 172.21.15.71:0/637627832 >> v1:172.21.15.137:6789/0 conn(0x7f9b58038de0 legacy=0x7f9b580cadf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:01:11.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.973+0000 7f9b5dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:01:11.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.973+0000 7f9b5dffb640 1 monclient: found mon.noname-a 2024-03-29T11:01:11.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.973+0000 7f9b5dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:11.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.973+0000 7f9b5dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:01:11.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.973+0000 7f9b5dffb640 1 -- 172.21.15.71:0/637627832 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b580356b0 con 0x7f9b58030680 2024-03-29T11:01:11.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.973+0000 7f9b5dffb640 10 monclient: _finish_auth 0 2024-03-29T11:01:11.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.973+0000 7f9b5dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:01:11.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.973+0000 7f9b5dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:11.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.973+0000 7f9b5dffb640 1 -- 172.21.15.71:0/637627832 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9b6004f730 con 0x7f9b58030680 2024-03-29T11:01:11.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.973+0000 7f9b656e4640 5 monclient: authenticate success, global_id 4260 2024-03-29T11:01:11.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.973+0000 7f9b656e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:11.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.973+0000 7f9b5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:11.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.973+0000 7f9b5dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:11.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.973+0000 7f9b5dffb640 10 monclient: dump: 2024-03-29T11:01:11.955 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:11.956 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:11.956 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:11.956 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:11.956 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:11.956 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:11.956 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:11.956 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:11.956 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:11.956 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:11.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.973+0000 7f9b656e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:11.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.974+0000 7f9b5dffb640 1 -- 172.21.15.71:0/637627832 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b600008d0 con 0x7f9b58030680 2024-03-29T11:01:11.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.974+0000 7f9b5dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:11.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.974+0000 7f9b5dffb640 1 -- 172.21.15.71:0/637627832 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9b600758b0 con 0x7f9b58030680 2024-03-29T11:01:11.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.974+0000 7f9b5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:11.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.974+0000 7f9b5dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:11.957 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.974+0000 7f9b5dffb640 10 monclient: dump: 2024-03-29T11:01:11.957 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:11.957 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:11.957 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:11.957 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:11.957 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:11.957 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:11.957 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:11.957 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:11.957 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:11.957 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:11.958 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.977+0000 7f9b656e4640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:11.958 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.977+0000 7f9b656e4640 4 set_mon_vals no callback set 2024-03-29T11:01:11.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.980+0000 7f9b656e4640 10 monclient: shutdown 2024-03-29T11:01:11.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.980+0000 7f9b656e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:11.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.980+0000 7f9b656e4640 1 -- 172.21.15.71:0/637627832 >> v1:172.21.15.71:6789/0 conn(0x7f9b58030680 legacy=0x7f9b580d3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:11.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.980+0000 7f9b656e4640 1 -- 172.21.15.71:0/637627832 shutdown_connections 2024-03-29T11:01:11.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.980+0000 7f9b656e4640 1 -- 172.21.15.71:0/637627832 >> 172.21.15.71:0/637627832 conn(0x7f9b58005c20 msgr2=0x7f9b58006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:11.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.980+0000 7f9b656e4640 1 -- 172.21.15.71:0/637627832 shutdown_connections 2024-03-29T11:01:11.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.980+0000 7f9b656e4640 1 -- 172.21.15.71:0/637627832 wait complete. 2024-03-29T11:01:11.961 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.981+0000 7f9b656e4640 10 monclient: build_initial_monmap 2024-03-29T11:01:11.962 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.981+0000 7f9b656e4640 10 monclient: monmap: 2024-03-29T11:01:11.962 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:01:11.962 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:11.962 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:01:11.962 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:01:11.962 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:11.962 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:11.962 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:01:11.962 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:01:11.962 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:01:11.962 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:11.962 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.981+0000 7f9b656e4640 1 librados: starting msgr at 2024-03-29T11:01:11.962 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.981+0000 7f9b656e4640 1 librados: starting objecter 2024-03-29T11:01:11.963 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.982+0000 7f9b656e4640 1 Processor -- start 2024-03-29T11:01:11.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.982+0000 7f9b656e4640 1 -- start start 2024-03-29T11:01:11.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.982+0000 7f9b656e4640 1 librados: setting wanted keys 2024-03-29T11:01:11.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.982+0000 7f9b656e4640 1 librados: calling monclient init 2024-03-29T11:01:11.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.982+0000 7f9b656e4640 10 monclient: init 2024-03-29T11:01:11.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.983+0000 7f9b656e4640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:11.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.983+0000 7f9b656e4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:01:11.965 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.983+0000 7f9b656e4640 10 monclient(hunting): picked mon.noname-c con 0x7f9b58030680 addr v1:172.21.15.71:6790/0 2024-03-29T11:01:11.965 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.983+0000 7f9b656e4640 10 monclient(hunting): picked mon.noname-b con 0x7f9b58038a00 addr v1:172.21.15.137:6789/0 2024-03-29T11:01:11.965 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.983+0000 7f9b656e4640 10 monclient(hunting): picked mon.noname-a con 0x7f9b5802d650 addr v1:172.21.15.71:6789/0 2024-03-29T11:01:11.965 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.983+0000 7f9b656e4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b5802fc40 con 0x7f9b5802d650 2024-03-29T11:01:11.965 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.983+0000 7f9b656e4640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b5802f110 con 0x7f9b58038a00 2024-03-29T11:01:11.965 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.983+0000 7f9b656e4640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b580c0d00 con 0x7f9b58030680 2024-03-29T11:01:11.965 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.983+0000 7f9b656e4640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:11.965 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.983+0000 7f9b656e4640 10 monclient(hunting): authenticate will time out at 828.241346s 2024-03-29T11:01:11.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.987+0000 7f9b5f7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f9b5802d650 0x7f9b580c05f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:50453/0 (socket says 172.21.15.71:50453) 2024-03-29T11:01:11.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.987+0000 7f9b5f7fe640 1 -- 172.21.15.71:0/1371023612 learned_addr learned my addr 172.21.15.71:0/1371023612 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:01:11.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.987+0000 7f9b477fe640 10 client.?.objecter ms_handle_connect 0x7f9b5802d650 2024-03-29T11:01:11.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.987+0000 7f9b477fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:11.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.987+0000 7f9b477fe640 1 -- 172.21.15.71:0/1371023612 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992185249 0 0) 0x7f9b5802fc40 con 0x7f9b5802d650 2024-03-29T11:01:11.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.987+0000 7f9b477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:01:11.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.987+0000 7f9b477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:11.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.987+0000 7f9b477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:11.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.987+0000 7f9b477fe640 10 monclient(hunting): my global_id is 4263 2024-03-29T11:01:11.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.987+0000 7f9b477fe640 1 -- 172.21.15.71:0/1371023612 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b4c004eb0 con 0x7f9b5802d650 2024-03-29T11:01:11.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.990+0000 7f9b477fe640 10 client.?.objecter ms_handle_connect 0x7f9b58030680 2024-03-29T11:01:11.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.990+0000 7f9b477fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:11.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.990+0000 7f9b477fe640 1 -- 172.21.15.71:0/1371023612 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2796420237 0 0) 0x7f9b580c0d00 con 0x7f9b58030680 2024-03-29T11:01:11.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.990+0000 7f9b477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:01:11.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.990+0000 7f9b477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:11.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.990+0000 7f9b477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:11.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.990+0000 7f9b477fe640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:01:11.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.990+0000 7f9b477fe640 1 -- 172.21.15.71:0/1371023612 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b5802fc40 con 0x7f9b58030680 2024-03-29T11:01:11.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b477fe640 1 -- 172.21.15.71:0/1371023612 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1093215520 0 0) 0x7f9b4c004eb0 con 0x7f9b5802d650 2024-03-29T11:01:11.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b477fe640 1 -- 172.21.15.71:0/1371023612 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b580c0d00 con 0x7f9b5802d650 2024-03-29T11:01:11.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b477fe640 1 -- 172.21.15.71:0/1371023612 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9b6004f400 con 0x7f9b5802d650 2024-03-29T11:01:11.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:11.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b477fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:11.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b477fe640 10 monclient(hunting): dump: 2024-03-29T11:01:11.975 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:11.975 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:11.975 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:11.975 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:11.975 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:11.976 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:11.976 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:11.976 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:11.976 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:11.976 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:11.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b477fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:11.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b477fe640 1 -- 172.21.15.71:0/1371023612 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2340428620 0 0) 0x7f9b5802fc40 con 0x7f9b58030680 2024-03-29T11:01:11.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b477fe640 1 -- 172.21.15.71:0/1371023612 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b4c004eb0 con 0x7f9b58030680 2024-03-29T11:01:11.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b477fe640 1 -- 172.21.15.71:0/1371023612 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9b50001930 con 0x7f9b58030680 2024-03-29T11:01:11.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:11.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b477fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:11.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b477fe640 10 monclient(hunting): dump: 2024-03-29T11:01:11.976 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:11.976 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:11.976 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:11.976 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:11.977 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:11.977 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:11.977 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:11.977 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:11.977 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:11.977 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:11.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b477fe640 1 -- 172.21.15.71:0/1371023612 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 558969920 0 0) 0x7f9b580c0d00 con 0x7f9b5802d650 2024-03-29T11:01:11.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b477fe640 1 -- 172.21.15.71:0/1371023612 >> v1:172.21.15.71:6790/0 conn(0x7f9b58030680 legacy=0x7f9b58161140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:11.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b477fe640 1 -- 172.21.15.71:0/1371023612 >> v1:172.21.15.137:6789/0 conn(0x7f9b58038a00 legacy=0x7f9b5802f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:11.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b477fe640 10 monclient: _finish_hunting 0 2024-03-29T11:01:11.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b477fe640 1 monclient: found mon.a 2024-03-29T11:01:11.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:11.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b477fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:01:11.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b477fe640 1 -- 172.21.15.71:0/1371023612 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b580c0e90 con 0x7f9b5802d650 2024-03-29T11:01:11.977 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b477fe640 10 monclient: _finish_auth 0 2024-03-29T11:01:11.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b477fe640 10 monclient: _check_auth_tickets 2024-03-29T11:01:11.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:11.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b656e4640 5 monclient: authenticate success, global_id 4263 2024-03-29T11:01:11.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b656e4640 10 monclient: _renew_subs 2024-03-29T11:01:11.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b656e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:01:11.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b656e4640 1 -- 172.21.15.71:0/1371023612 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b5802d170 con 0x7f9b5802d650 2024-03-29T11:01:11.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b656e4640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:11.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b656e4640 10 monclient: _renew_subs 2024-03-29T11:01:11.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.991+0000 7f9b656e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:01:11.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.992+0000 7f9b656e4640 1 -- 172.21.15.71:0/1371023612 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b580c8200 con 0x7f9b5802d650 2024-03-29T11:01:11.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.992+0000 7f9b656e4640 1 librados: init done 2024-03-29T11:01:11.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.992+0000 7f9b477fe640 1 -- 172.21.15.71:0/1371023612 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b6004fdd0 con 0x7f9b5802d650 2024-03-29T11:01:11.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.992+0000 7f9b477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:11.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.992+0000 7f9b477fe640 1 -- 172.21.15.71:0/1371023612 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9b60067bf0 con 0x7f9b5802d650 2024-03-29T11:01:11.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.992+0000 7f9b477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:11.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.992+0000 7f9b477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:11.982 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.992+0000 7f9b477fe640 10 monclient: dump: 2024-03-29T11:01:11.982 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:11.982 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:11.982 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:11.982 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:11.982 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:11.982 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:11.982 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:11.983 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:11.983 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:11.983 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:11.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.993+0000 7f9b656e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:11.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.993+0000 7f9b656e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:11.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.994+0000 7f9b656e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:01:11.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.994+0000 7f9b656e4640 1 -- 172.21.15.71:0/1371023612 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b5802ef70 con 0x7f9b5802d650 2024-03-29T11:01:11.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.994+0000 7f9b5cff9640 4 set_mon_vals no callback set 2024-03-29T11:01:11.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.994+0000 7f9b477fe640 1 -- 172.21.15.71:0/1371023612 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f9b60067d50 con 0x7f9b5802d650 2024-03-29T11:01:11.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.995+0000 7f9b477fe640 1 -- 172.21.15.71:0/1371023612 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f9b600ff5b0 con 0x7f9b5802d650 2024-03-29T11:01:11.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.995+0000 7f9b477fe640 10 client.4263.objecter ms_dispatch 0x7f9b60000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:11.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.995+0000 7f9b477fe640 3 client.4263.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:11.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.995+0000 7f9b477fe640 3 client.4263.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:11.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.995+0000 7f9b477fe640 20 client.4263.objecter dump_active .. 0 homeless 2024-03-29T11:01:11.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.996+0000 7f9b5effd640 10 monclient: get_auth_request con 0x7f9b4c06dd70 auth_method 0 2024-03-29T11:01:11.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:11.996+0000 7f9b477fe640 10 client.4263.objecter ms_handle_connect 0x7f9b4c06dd70 2024-03-29T11:01:11.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:12.004+0000 7f9b477fe640 1 -- 172.21.15.71:0/1371023612 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9b5802ef70 con 0x7f9b5802d650 2024-03-29T11:01:11.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:12.004+0000 7f9b477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:11.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:12.004+0000 7f9b477fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:12.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:12.333+0000 7f9b656e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:12.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:12.333+0000 7f9b656e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:12.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:12.333+0000 7f9b656e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:01:12.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:12.333+0000 7f9b656e4640 1 -- 172.21.15.71:0/1371023612 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b5802ef70 con 0x7f9b5802d650 2024-03-29T11:01:12.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:12.334+0000 7f9b477fe640 1 -- 172.21.15.71:0/1371023612 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f9b5802ef70 con 0x7f9b5802d650 2024-03-29T11:01:12.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:12.334+0000 7f9b477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:12.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:12.334+0000 7f9b477fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:12.314 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:01:12.314 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:12.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:12.337+0000 7f9b656e4640 10 librados: watch_flush enter 2024-03-29T11:01:12.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:12.337+0000 7f9b656e4640 10 librados: watch_flush exit 2024-03-29T11:01:12.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:12.337+0000 7f9b656e4640 20 client.4263.objecter shutdown clearing up homeless session... 2024-03-29T11:01:12.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:12.337+0000 7f9b656e4640 10 client.4263.objecter successfully canceled tick 2024-03-29T11:01:12.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:12.337+0000 7f9b656e4640 1 -- 172.21.15.71:0/1371023612 >> v1:172.21.15.137:6816/973332665 conn(0x7f9b4c06dd70 legacy=0x7f9b4c070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:12.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:12.337+0000 7f9b656e4640 10 monclient: shutdown 2024-03-29T11:01:12.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:12.337+0000 7f9b656e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:12.318 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:12.337+0000 7f9b656e4640 1 -- 172.21.15.71:0/1371023612 >> v1:172.21.15.71:6789/0 conn(0x7f9b5802d650 legacy=0x7f9b580c05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:12.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:12.337+0000 7f9b656e4640 1 -- 172.21.15.71:0/1371023612 shutdown_connections 2024-03-29T11:01:12.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:12.337+0000 7f9b656e4640 1 -- 172.21.15.71:0/1371023612 >> 172.21.15.71:0/1371023612 conn(0x7f9b58005c20 msgr2=0x7f9b580340b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:12.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:12.341+0000 7f9b656e4640 1 -- 172.21.15.71:0/1371023612 shutdown_connections 2024-03-29T11:01:12.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:12.341+0000 7f9b656e4640 1 -- 172.21.15.71:0/1371023612 wait complete. 2024-03-29T11:01:12.321 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:12.341+0000 7f9b656e4640 1 librados: shutdown 2024-03-29T11:01:12.341 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:19.148 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:19.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.312+0000 7f9ad82dd640 10 monclient: get_monmap_and_config 2024-03-29T11:01:19.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.312+0000 7f9ad82dd640 10 monclient: build_initial_monmap 2024-03-29T11:01:19.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.312+0000 7f9ad82dd640 10 monclient: monmap: 2024-03-29T11:01:19.293 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:01:19.293 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:19.293 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:01:19.313762+0000 2024-03-29T11:01:19.293 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:01:19.313762+0000 2024-03-29T11:01:19.293 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:19.293 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:19.293 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:01:19.293 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:01:19.293 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:01:19.293 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:19.294 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.313+0000 7f9ad82dd640 1 Processor -- start 2024-03-29T11:01:19.294 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.313+0000 7f9ad82dd640 1 -- start start 2024-03-29T11:01:19.294 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.313+0000 7f9ad82dd640 10 monclient: init 2024-03-29T11:01:19.297 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.314+0000 7f9ad82dd640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:19.297 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.314+0000 7f9ad82dd640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:01:19.297 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.314+0000 7f9ad82dd640 10 monclient(hunting): picked mon.noname-b con 0x7f9ad01457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:01:19.297 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.314+0000 7f9ad82dd640 10 monclient(hunting): picked mon.noname-c con 0x7f9ad013f650 addr v1:172.21.15.137:6789/0 2024-03-29T11:01:19.297 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.314+0000 7f9ad82dd640 10 monclient(hunting): picked mon.noname-a con 0x7f9ad00a9d70 addr v1:172.21.15.71:6789/0 2024-03-29T11:01:19.297 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.314+0000 7f9ad82dd640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad00aaef0 con 0x7f9ad00a9d70 2024-03-29T11:01:19.298 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.314+0000 7f9ad82dd640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad0145bd0 con 0x7f9ad013f650 2024-03-29T11:01:19.298 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.314+0000 7f9ad82dd640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad0145eb0 con 0x7f9ad01457f0 2024-03-29T11:01:19.298 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.314+0000 7f9ad82dd640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:19.298 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.314+0000 7f9ad82dd640 10 monclient(hunting): authenticate will time out at 835.573011s 2024-03-29T11:01:19.298 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.318+0000 7f9ad687a640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f9ad00a9d70 0x7f9ad014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:44455/0 (socket says 172.21.15.71:44455) 2024-03-29T11:01:19.298 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.318+0000 7f9ad687a640 1 -- 172.21.15.71:0/2800743065 learned_addr learned my addr 172.21.15.71:0/2800743065 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:01:19.298 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.318+0000 7f9ad5077640 1 -- 172.21.15.71:0/2800743065 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569994802 0 0) 0x7f9ad00aaef0 con 0x7f9ad00a9d70 2024-03-29T11:01:19.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.318+0000 7f9ad5077640 10 monclient(hunting): _negotiate 2024-03-29T11:01:19.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.318+0000 7f9ad5077640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:19.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.318+0000 7f9ad5077640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:19.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.318+0000 7f9ad5077640 10 monclient(hunting): my global_id is 4266 2024-03-29T11:01:19.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.318+0000 7f9ad5077640 1 -- 172.21.15.71:0/2800743065 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9abc003520 con 0x7f9ad00a9d70 2024-03-29T11:01:19.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.319+0000 7f9ad5077640 1 -- 172.21.15.71:0/2800743065 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488146514 0 0) 0x7f9ad0145eb0 con 0x7f9ad01457f0 2024-03-29T11:01:19.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.319+0000 7f9ad5077640 10 monclient(hunting): _negotiate 2024-03-29T11:01:19.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.319+0000 7f9ad5077640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:19.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.319+0000 7f9ad5077640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:19.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.319+0000 7f9ad5077640 10 monclient(hunting): my global_id is 4256 2024-03-29T11:01:19.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.319+0000 7f9ad5077640 1 -- 172.21.15.71:0/2800743065 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ad00aaef0 con 0x7f9ad01457f0 2024-03-29T11:01:19.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.320+0000 7f9ad5077640 1 -- 172.21.15.71:0/2800743065 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 561064446 0 0) 0x7f9ad00aaef0 con 0x7f9ad01457f0 2024-03-29T11:01:19.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.320+0000 7f9ad5077640 1 -- 172.21.15.71:0/2800743065 >> v1:172.21.15.137:6789/0 conn(0x7f9ad013f650 legacy=0x7f9ad00b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:01:19.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.321+0000 7f9ad5077640 1 -- 172.21.15.71:0/2800743065 >> v1:172.21.15.71:6789/0 conn(0x7f9ad00a9d70 legacy=0x7f9ad014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:19.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.321+0000 7f9ad5077640 10 monclient: _finish_hunting 0 2024-03-29T11:01:19.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.321+0000 7f9ad5077640 1 monclient: found mon.noname-b 2024-03-29T11:01:19.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.321+0000 7f9ad5077640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:19.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.321+0000 7f9ad5077640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:01:19.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.321+0000 7f9ad5077640 1 -- 172.21.15.71:0/2800743065 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad0146340 con 0x7f9ad01457f0 2024-03-29T11:01:19.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.321+0000 7f9ad5077640 10 monclient: _finish_auth 0 2024-03-29T11:01:19.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.321+0000 7f9ad5077640 10 monclient: _check_auth_tickets 2024-03-29T11:01:19.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.321+0000 7f9ad5077640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:19.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.321+0000 7f9ad5077640 1 -- 172.21.15.71:0/2800743065 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9acc001670 con 0x7f9ad01457f0 2024-03-29T11:01:19.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.321+0000 7f9ad5077640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:19.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.321+0000 7f9ad5077640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:19.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.321+0000 7f9ad5077640 10 monclient: dump: 2024-03-29T11:01:19.304 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:19.304 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:19.304 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:19.304 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:19.304 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:19.304 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:19.304 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:19.304 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:19.304 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:19.304 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:19.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.321+0000 7f9ad5077640 1 -- 172.21.15.71:0/2800743065 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9acc0017d0 con 0x7f9ad01457f0 2024-03-29T11:01:19.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.321+0000 7f9ad5077640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:19.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.321+0000 7f9ad5077640 1 -- 172.21.15.71:0/2800743065 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9acc001be0 con 0x7f9ad01457f0 2024-03-29T11:01:19.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.321+0000 7f9ad5077640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:19.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.321+0000 7f9ad5077640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:19.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.321+0000 7f9ad5077640 10 monclient: dump: 2024-03-29T11:01:19.305 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:19.305 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:19.305 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:19.305 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:19.305 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:19.305 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:19.305 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:19.305 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:19.305 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:19.305 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:19.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.322+0000 7f9ad82dd640 5 monclient: authenticate success, global_id 4256 2024-03-29T11:01:19.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.322+0000 7f9ad82dd640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:19.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.322+0000 7f9ad82dd640 4 set_mon_vals no callback set 2024-03-29T11:01:19.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.323+0000 7f9ad82dd640 10 monclient: shutdown 2024-03-29T11:01:19.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.323+0000 7f9ad82dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:19.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.323+0000 7f9ad82dd640 1 -- 172.21.15.71:0/2800743065 >> v1:172.21.15.71:6790/0 conn(0x7f9ad01457f0 legacy=0x7f9ad013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:19.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.325+0000 7f9ad82dd640 1 -- 172.21.15.71:0/2800743065 shutdown_connections 2024-03-29T11:01:19.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.325+0000 7f9ad82dd640 1 -- 172.21.15.71:0/2800743065 >> 172.21.15.71:0/2800743065 conn(0x7f9ad0066a00 msgr2=0x7f9ad0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:19.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.325+0000 7f9ad82dd640 1 -- 172.21.15.71:0/2800743065 shutdown_connections 2024-03-29T11:01:19.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.325+0000 7f9ad82dd640 1 -- 172.21.15.71:0/2800743065 wait complete. 2024-03-29T11:01:19.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.326+0000 7f9ad82dd640 10 monclient: build_initial_monmap 2024-03-29T11:01:19.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.326+0000 7f9ad82dd640 10 monclient: monmap: 2024-03-29T11:01:19.308 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:01:19.308 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:19.308 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:01:19.308 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:01:19.308 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:19.309 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:19.309 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:01:19.309 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:01:19.309 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:01:19.309 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:19.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.327+0000 7f9ad82dd640 1 librados: starting msgr at 2024-03-29T11:01:19.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.327+0000 7f9ad82dd640 1 librados: starting objecter 2024-03-29T11:01:19.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.327+0000 7f9ad82dd640 1 Processor -- start 2024-03-29T11:01:19.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.327+0000 7f9ad82dd640 1 -- start start 2024-03-29T11:01:19.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.327+0000 7f9ad82dd640 1 librados: setting wanted keys 2024-03-29T11:01:19.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.327+0000 7f9ad82dd640 1 librados: calling monclient init 2024-03-29T11:01:19.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.327+0000 7f9ad82dd640 10 monclient: init 2024-03-29T11:01:19.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.328+0000 7f9ad82dd640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:19.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.328+0000 7f9ad82dd640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:01:19.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.329+0000 7f9ad82dd640 10 monclient(hunting): picked mon.noname-c con 0x7f9ad00a9d70 addr v1:172.21.15.71:6790/0 2024-03-29T11:01:19.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.329+0000 7f9ad82dd640 10 monclient(hunting): picked mon.noname-a con 0x7f9ad013f650 addr v1:172.21.15.71:6789/0 2024-03-29T11:01:19.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.329+0000 7f9ad82dd640 10 monclient(hunting): picked mon.noname-b con 0x7f9ad01457f0 addr v1:172.21.15.137:6789/0 2024-03-29T11:01:19.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.329+0000 7f9ad82dd640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad0135870 con 0x7f9ad013f650 2024-03-29T11:01:19.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.329+0000 7f9ad82dd640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad0141ec0 con 0x7f9ad01457f0 2024-03-29T11:01:19.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.329+0000 7f9ad82dd640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad013b850 con 0x7f9ad00a9d70 2024-03-29T11:01:19.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.329+0000 7f9ad82dd640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:19.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.329+0000 7f9ad82dd640 10 monclient(hunting): authenticate will time out at 835.587588s 2024-03-29T11:01:19.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.332+0000 7f9ad6079640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f9ad00a9d70 0x7f9ad01d75e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:45535/0 (socket says 172.21.15.71:45535) 2024-03-29T11:01:19.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.332+0000 7f9ad6079640 1 -- 172.21.15.71:0/384836785 learned_addr learned my addr 172.21.15.71:0/384836785 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:01:19.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.333+0000 7f9ac6ffd640 10 client.?.objecter ms_handle_connect 0x7f9ad00a9d70 2024-03-29T11:01:19.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.333+0000 7f9ac6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:19.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.333+0000 7f9ac6ffd640 10 client.?.objecter ms_handle_connect 0x7f9ad013f650 2024-03-29T11:01:19.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.333+0000 7f9ac6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:19.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.333+0000 7f9ac6ffd640 1 -- 172.21.15.71:0/384836785 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1085660634 0 0) 0x7f9ad013b850 con 0x7f9ad00a9d70 2024-03-29T11:01:19.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.333+0000 7f9ac6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:19.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.333+0000 7f9ac6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:19.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.333+0000 7f9ac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:19.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.333+0000 7f9ac6ffd640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:01:19.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.333+0000 7f9ac6ffd640 1 -- 172.21.15.71:0/384836785 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ab4004eb0 con 0x7f9ad00a9d70 2024-03-29T11:01:19.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.334+0000 7f9ac6ffd640 1 -- 172.21.15.71:0/384836785 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879097263 0 0) 0x7f9ad0135870 con 0x7f9ad013f650 2024-03-29T11:01:19.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.334+0000 7f9ac6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:19.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.334+0000 7f9ac6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:19.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.334+0000 7f9ac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:19.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.334+0000 7f9ac6ffd640 10 monclient(hunting): my global_id is 4269 2024-03-29T11:01:19.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.334+0000 7f9ac6ffd640 1 -- 172.21.15.71:0/384836785 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ab40051d0 con 0x7f9ad013f650 2024-03-29T11:01:19.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.334+0000 7f9ac6ffd640 1 -- 172.21.15.71:0/384836785 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 841296797 0 0) 0x7f9ab4004eb0 con 0x7f9ad00a9d70 2024-03-29T11:01:19.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.334+0000 7f9ac6ffd640 1 -- 172.21.15.71:0/384836785 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ab4005640 con 0x7f9ad00a9d70 2024-03-29T11:01:19.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.334+0000 7f9ac6ffd640 1 -- 172.21.15.71:0/384836785 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9acc001670 con 0x7f9ad00a9d70 2024-03-29T11:01:19.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.334+0000 7f9ac6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:19.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.334+0000 7f9ac6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:19.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.334+0000 7f9ac6ffd640 10 monclient(hunting): dump: 2024-03-29T11:01:19.316 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:19.316 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:19.317 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:19.317 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:19.317 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:19.317 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:19.317 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:19.317 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:19.317 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:19.317 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:19.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.334+0000 7f9ac6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:19.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.335+0000 7f9ac6ffd640 1 -- 172.21.15.71:0/384836785 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2142460323 0 0) 0x7f9ab40051d0 con 0x7f9ad013f650 2024-03-29T11:01:19.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.335+0000 7f9ac6ffd640 1 -- 172.21.15.71:0/384836785 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ab4005ce0 con 0x7f9ad013f650 2024-03-29T11:01:19.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.335+0000 7f9ac6ffd640 1 -- 172.21.15.71:0/384836785 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9ac0001830 con 0x7f9ad013f650 2024-03-29T11:01:19.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.335+0000 7f9ac6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:19.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.335+0000 7f9ac6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:19.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.335+0000 7f9ac6ffd640 10 monclient(hunting): dump: 2024-03-29T11:01:19.318 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:19.318 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:19.318 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:19.318 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:19.318 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:19.318 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:19.318 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:19.318 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:19.318 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:19.318 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:19.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.336+0000 7f9ac6ffd640 1 -- 172.21.15.71:0/384836785 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1372455651 0 0) 0x7f9ab4005640 con 0x7f9ad00a9d70 2024-03-29T11:01:19.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.336+0000 7f9ac6ffd640 1 -- 172.21.15.71:0/384836785 >> v1:172.21.15.137:6789/0 conn(0x7f9ad01457f0 legacy=0x7f9ad0135160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:01:19.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.336+0000 7f9ac6ffd640 1 -- 172.21.15.71:0/384836785 >> v1:172.21.15.71:6789/0 conn(0x7f9ad013f650 legacy=0x7f9ad0134a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:19.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.336+0000 7f9ac6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:01:19.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.336+0000 7f9ac6ffd640 1 monclient: found mon.c 2024-03-29T11:01:19.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.336+0000 7f9ac6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:19.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.336+0000 7f9ac6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:01:19.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.336+0000 7f9ac6ffd640 1 -- 172.21.15.71:0/384836785 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad0140f80 con 0x7f9ad00a9d70 2024-03-29T11:01:19.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.336+0000 7f9ac6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:01:19.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.336+0000 7f9ac6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:01:19.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.336+0000 7f9ac6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:19.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.337+0000 7f9ad82dd640 5 monclient: authenticate success, global_id 4259 2024-03-29T11:01:19.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.337+0000 7f9ad82dd640 10 monclient: _renew_subs 2024-03-29T11:01:19.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.337+0000 7f9ad82dd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:01:19.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.337+0000 7f9ad82dd640 1 -- 172.21.15.71:0/384836785 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ad01d7cf0 con 0x7f9ad00a9d70 2024-03-29T11:01:19.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.337+0000 7f9ad82dd640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:19.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.337+0000 7f9ad82dd640 10 monclient: _renew_subs 2024-03-29T11:01:19.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.337+0000 7f9ad82dd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:01:19.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.337+0000 7f9ad82dd640 1 -- 172.21.15.71:0/384836785 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad0142b10 con 0x7f9ad00a9d70 2024-03-29T11:01:19.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.337+0000 7f9ad82dd640 1 librados: init done 2024-03-29T11:01:19.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.339+0000 7f9ac6ffd640 1 -- 172.21.15.71:0/384836785 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9acc0038a0 con 0x7f9ad00a9d70 2024-03-29T11:01:19.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.339+0000 7f9ac6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:19.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.339+0000 7f9ac6ffd640 1 -- 172.21.15.71:0/384836785 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9acc001670 con 0x7f9ad00a9d70 2024-03-29T11:01:19.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.339+0000 7f9ac6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:19.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.339+0000 7f9ac6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:19.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.339+0000 7f9ac6ffd640 10 monclient: dump: 2024-03-29T11:01:19.325 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:19.325 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:19.325 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:19.325 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:19.325 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:19.325 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:19.325 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:19.325 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:19.325 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:19.325 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:19.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.339+0000 7f9ac7fff640 4 set_mon_vals no callback set 2024-03-29T11:01:19.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.339+0000 7f9ad82dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:19.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.340+0000 7f9ad82dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:19.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.340+0000 7f9ad82dd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:01:19.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.340+0000 7f9ad82dd640 1 -- 172.21.15.71:0/384836785 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ab8005080 con 0x7f9ad00a9d70 2024-03-29T11:01:19.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.340+0000 7f9ac6ffd640 1 -- 172.21.15.71:0/384836785 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f9acc003bd0 con 0x7f9ad00a9d70 2024-03-29T11:01:19.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.342+0000 7f9ac6ffd640 1 -- 172.21.15.71:0/384836785 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f9acc054670 con 0x7f9ad00a9d70 2024-03-29T11:01:19.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.342+0000 7f9ac6ffd640 10 client.4259.objecter ms_dispatch 0x7f9ad0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:19.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.342+0000 7f9ac6ffd640 3 client.4259.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:19.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.342+0000 7f9ac6ffd640 3 client.4259.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:19.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.342+0000 7f9ac6ffd640 20 client.4259.objecter dump_active .. 0 homeless 2024-03-29T11:01:19.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.342+0000 7f9ad5878640 10 monclient: get_auth_request con 0x7f9ab406e710 auth_method 0 2024-03-29T11:01:19.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.345+0000 7f9ac6ffd640 10 client.4259.objecter ms_handle_connect 0x7f9ab406e710 2024-03-29T11:01:19.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.352+0000 7f9ac6ffd640 1 -- 172.21.15.71:0/384836785 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9acc050910 con 0x7f9ad00a9d70 2024-03-29T11:01:19.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.352+0000 7f9ac6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:19.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.352+0000 7f9ac6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:19.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.673+0000 7f9ad82dd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:19.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.673+0000 7f9ad82dd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:19.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.673+0000 7f9ad82dd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:01:19.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.673+0000 7f9ad82dd640 1 -- 172.21.15.71:0/384836785 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ab8005330 con 0x7f9ad00a9d70 2024-03-29T11:01:19.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.673+0000 7f9ac6ffd640 1 -- 172.21.15.71:0/384836785 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f9acc050910 con 0x7f9ad00a9d70 2024-03-29T11:01:19.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.673+0000 7f9ac6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:19.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.673+0000 7f9ac6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:19.656 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:01:19.656 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:19.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.681+0000 7f9ad82dd640 10 librados: watch_flush enter 2024-03-29T11:01:19.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.681+0000 7f9ad82dd640 10 librados: watch_flush exit 2024-03-29T11:01:19.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.681+0000 7f9ad82dd640 20 client.4259.objecter shutdown clearing up homeless session... 2024-03-29T11:01:19.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.681+0000 7f9ad82dd640 10 client.4259.objecter successfully canceled tick 2024-03-29T11:01:19.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.682+0000 7f9ad82dd640 1 -- 172.21.15.71:0/384836785 >> v1:172.21.15.137:6816/973332665 conn(0x7f9ab406e710 legacy=0x7f9ab4070bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:19.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.682+0000 7f9ad82dd640 10 monclient: shutdown 2024-03-29T11:01:19.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.682+0000 7f9ad82dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:19.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.682+0000 7f9ad82dd640 1 -- 172.21.15.71:0/384836785 >> v1:172.21.15.71:6790/0 conn(0x7f9ad00a9d70 legacy=0x7f9ad01d75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:19.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.682+0000 7f9ad82dd640 1 -- 172.21.15.71:0/384836785 shutdown_connections 2024-03-29T11:01:19.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.682+0000 7f9ad82dd640 1 -- 172.21.15.71:0/384836785 >> 172.21.15.71:0/384836785 conn(0x7f9ad0066a00 msgr2=0x7f9ad00b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:19.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.682+0000 7f9ad82dd640 1 -- 172.21.15.71:0/384836785 shutdown_connections 2024-03-29T11:01:19.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.683+0000 7f9ad82dd640 1 -- 172.21.15.71:0/384836785 wait complete. 2024-03-29T11:01:19.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:19.683+0000 7f9ad82dd640 1 librados: shutdown 2024-03-29T11:01:19.679 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:26.485 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:26.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.662+0000 7fd303338640 10 monclient: get_monmap_and_config 2024-03-29T11:01:26.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.662+0000 7fd303338640 10 monclient: build_initial_monmap 2024-03-29T11:01:26.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.662+0000 7fd303338640 10 monclient: monmap: 2024-03-29T11:01:26.642 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:01:26.643 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:26.643 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:01:26.663252+0000 2024-03-29T11:01:26.643 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:01:26.663252+0000 2024-03-29T11:01:26.643 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:26.643 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:26.643 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:01:26.643 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:01:26.643 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:01:26.643 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:26.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.663+0000 7fd303338640 1 Processor -- start 2024-03-29T11:01:26.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.663+0000 7fd303338640 1 -- start start 2024-03-29T11:01:26.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.663+0000 7fd303338640 10 monclient: init 2024-03-29T11:01:26.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.664+0000 7fd303338640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:26.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.664+0000 7fd303338640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:01:26.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.664+0000 7fd303338640 10 monclient(hunting): picked mon.noname-a con 0x7fd2fc0b01a0 addr v1:172.21.15.71:6789/0 2024-03-29T11:01:26.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.664+0000 7fd303338640 10 monclient(hunting): picked mon.noname-b con 0x7fd2fc0af010 addr v1:172.21.15.71:6790/0 2024-03-29T11:01:26.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.664+0000 7fd303338640 10 monclient(hunting): picked mon.noname-c con 0x7fd2fc0ac5a0 addr v1:172.21.15.137:6789/0 2024-03-29T11:01:26.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.664+0000 7fd303338640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2fc0ae8a0 con 0x7fd2fc0b01a0 2024-03-29T11:01:26.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.664+0000 7fd303338640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2fc0ab1d0 con 0x7fd2fc0ac5a0 2024-03-29T11:01:26.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.664+0000 7fd303338640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2fc0ab4b0 con 0x7fd2fc0af010 2024-03-29T11:01:26.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.664+0000 7fd303338640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:26.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.664+0000 7fd303338640 10 monclient(hunting): authenticate will time out at 842.922708s 2024-03-29T11:01:26.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.670+0000 7fd3010d4640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fd2fc0b01a0 0x7fd2fc0abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:50029/0 (socket says 172.21.15.71:50029) 2024-03-29T11:01:26.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.670+0000 7fd3010d4640 1 -- 172.21.15.71:0/3030139765 learned_addr learned my addr 172.21.15.71:0/3030139765 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:01:26.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.670+0000 7fd2f3fff640 1 -- 172.21.15.71:0/3030139765 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814389951 0 0) 0x7fd2fc0ae8a0 con 0x7fd2fc0b01a0 2024-03-29T11:01:26.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.670+0000 7fd2f3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:26.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.670+0000 7fd2f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:26.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.670+0000 7fd2f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:26.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.670+0000 7fd2f3fff640 10 monclient(hunting): my global_id is 4272 2024-03-29T11:01:26.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.670+0000 7fd2f3fff640 1 -- 172.21.15.71:0/3030139765 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2e8003520 con 0x7fd2fc0b01a0 2024-03-29T11:01:26.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.672+0000 7fd2f3fff640 1 -- 172.21.15.71:0/3030139765 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 991713058 0 0) 0x7fd2e8003520 con 0x7fd2fc0b01a0 2024-03-29T11:01:26.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.672+0000 7fd2f3fff640 1 -- 172.21.15.71:0/3030139765 >> v1:172.21.15.71:6790/0 conn(0x7fd2fc0af010 legacy=0x7fd2fc149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:01:26.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.672+0000 7fd2f3fff640 1 -- 172.21.15.71:0/3030139765 >> v1:172.21.15.137:6789/0 conn(0x7fd2fc0ac5a0 legacy=0x7fd2fc151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:26.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.672+0000 7fd2f3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:01:26.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.672+0000 7fd2f3fff640 1 monclient: found mon.noname-a 2024-03-29T11:01:26.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.672+0000 7fd2f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:26.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.672+0000 7fd2f3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:01:26.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.672+0000 7fd2f3fff640 1 -- 172.21.15.71:0/3030139765 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2fc0ab790 con 0x7fd2fc0b01a0 2024-03-29T11:01:26.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.672+0000 7fd2f3fff640 10 monclient: _finish_auth 0 2024-03-29T11:01:26.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.672+0000 7fd2f3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:01:26.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.672+0000 7fd2f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:26.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.672+0000 7fd2f3fff640 1 -- 172.21.15.71:0/3030139765 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fd2ec001690 con 0x7fd2fc0b01a0 2024-03-29T11:01:26.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.672+0000 7fd2f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:26.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.672+0000 7fd2f3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:26.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.672+0000 7fd2f3fff640 10 monclient: dump: 2024-03-29T11:01:26.653 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:26.653 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:26.653 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:26.653 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:26.653 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:26.653 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:26.654 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:26.654 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:26.654 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:26.654 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:26.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.672+0000 7fd3010d4640 1 -- 172.21.15.71:0/3030139765 >> v1:172.21.15.71:6789/0 conn(0x7fd2fc0b01a0 legacy=0x7fd2fc0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-29T11:01:26.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.672+0000 7fd3010d4640 1 -- 172.21.15.71:0/3030139765 >> v1:172.21.15.71:6789/0 conn(0x7fd2fc0b01a0 legacy=0x7fd2fc0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:01:26.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.672+0000 7fd3010d4640 1 --1- 172.21.15.71:0/3030139765 >> v1:172.21.15.71:6789/0 conn(0x7fd2fc0b01a0 0x7fd2fc0abe90 :-1 s=OPENED pgs=87862 cs=1 l=1).handle_message read tag failed 2024-03-29T11:01:26.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.672+0000 7fd3010d4640 1 --1- 172.21.15.71:0/3030139765 >> v1:172.21.15.71:6789/0 conn(0x7fd2fc0b01a0 0x7fd2fc0abe90 :-1 s=OPENED pgs=87862 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:01:26.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.672+0000 7fd2f3fff640 10 monclient: ms_handle_reset current mon v1:172.21.15.71:6789/0 2024-03-29T11:01:26.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.672+0000 7fd2f3fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:26.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.672+0000 7fd2f3fff640 1 -- 172.21.15.71:0/3030139765 >> v1:172.21.15.71:6789/0 conn(0x7fd2fc0b01a0 legacy=0x7fd2fc0abe90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:01:26.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.672+0000 7fd2f3fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:01:26.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.673+0000 7fd2f3fff640 10 monclient(hunting): picked mon.b con 0x7fd2fc0ac5a0 addr v1:172.21.15.137:6789/0 2024-03-29T11:01:26.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.673+0000 7fd2f3fff640 10 monclient(hunting): picked mon.c con 0x7fd2e8004340 addr v1:172.21.15.71:6790/0 2024-03-29T11:01:26.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.673+0000 7fd2f3fff640 10 monclient(hunting): picked mon.a con 0x7fd2e800b300 addr v1:172.21.15.71:6789/0 2024-03-29T11:01:26.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.673+0000 7fd2f3fff640 1 -- 172.21.15.71:0/3030139765 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd2e8003520 con 0x7fd2e800b300 2024-03-29T11:01:26.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.673+0000 7fd2f3fff640 1 -- 172.21.15.71:0/3030139765 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd2fc0ab4b0 con 0x7fd2fc0ac5a0 2024-03-29T11:01:26.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.673+0000 7fd2f3fff640 1 -- 172.21.15.71:0/3030139765 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd2fc0ae8a0 con 0x7fd2e8004340 2024-03-29T11:01:26.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.673+0000 7fd2f3fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:26.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.673+0000 7fd2f3fff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.71:6789/0 2024-03-29T11:01:26.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.680+0000 7fd2f3fff640 1 -- 172.21.15.71:0/3030139765 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 244252382 0 0) 0x7fd2fc0ab4b0 con 0x7fd2fc0ac5a0 2024-03-29T11:01:26.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.680+0000 7fd2f3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:26.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.680+0000 7fd2f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:26.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.680+0000 7fd2f3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:01:26.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.680+0000 7fd2f3fff640 1 -- 172.21.15.71:0/3030139765 --> v1:172.21.15.137:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fd2e8010110 con 0x7fd2fc0ac5a0 2024-03-29T11:01:26.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.680+0000 7fd2f3fff640 1 -- 172.21.15.71:0/3030139765 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 219782925 0 0) 0x7fd2e8010110 con 0x7fd2fc0ac5a0 2024-03-29T11:01:26.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.680+0000 7fd2f3fff640 1 -- 172.21.15.71:0/3030139765 >> v1:172.21.15.71:6790/0 conn(0x7fd2e8004340 legacy=0x7fd2e80099b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:01:26.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.680+0000 7fd2f3fff640 1 -- 172.21.15.71:0/3030139765 >> v1:172.21.15.71:6789/0 conn(0x7fd2e800b300 legacy=0x7fd2e800d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:01:26.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.680+0000 7fd2f3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:01:26.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.680+0000 7fd2f3fff640 1 monclient: found mon.b 2024-03-29T11:01:26.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.680+0000 7fd2f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:26.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.680+0000 7fd2f3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:01:26.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.680+0000 7fd2f3fff640 1 -- 172.21.15.71:0/3030139765 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7fd2e8012320 con 0x7fd2fc0ac5a0 2024-03-29T11:01:26.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.680+0000 7fd2f3fff640 10 monclient: _finish_auth 0 2024-03-29T11:01:26.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.680+0000 7fd2f3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:01:26.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.680+0000 7fd2f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:26.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.680+0000 7fd2f3fff640 1 -- 172.21.15.71:0/3030139765 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fd2ec004a40 con 0x7fd2fc0ac5a0 2024-03-29T11:01:26.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.680+0000 7fd2f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:26.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.680+0000 7fd2f3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:26.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.680+0000 7fd2f3fff640 10 monclient: dump: 2024-03-29T11:01:26.661 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:26.661 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:26.661 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:26.662 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:26.662 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:26.662 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:26.662 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:26.662 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:26.662 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:26.662 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:26.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.681+0000 7fd2f3fff640 1 -- 172.21.15.71:0/3030139765 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2ec004ba0 con 0x7fd2fc0ac5a0 2024-03-29T11:01:26.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.681+0000 7fd2f3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:26.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.682+0000 7fd303338640 5 monclient: authenticate success, global_id 4272 2024-03-29T11:01:26.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.682+0000 7fd303338640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:26.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.682+0000 7fd303338640 4 set_mon_vals no callback set 2024-03-29T11:01:26.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.683+0000 7fd303338640 10 monclient: shutdown 2024-03-29T11:01:26.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.683+0000 7fd303338640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:26.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.683+0000 7fd303338640 1 -- 172.21.15.71:0/3030139765 >> v1:172.21.15.137:6789/0 conn(0x7fd2fc0ac5a0 legacy=0x7fd2e8003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:26.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.684+0000 7fd303338640 1 -- 172.21.15.71:0/3030139765 shutdown_connections 2024-03-29T11:01:26.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.684+0000 7fd303338640 1 -- 172.21.15.71:0/3030139765 >> 172.21.15.71:0/3030139765 conn(0x7fd2fc06b660 msgr2=0x7fd2fc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:26.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.684+0000 7fd303338640 1 -- 172.21.15.71:0/3030139765 shutdown_connections 2024-03-29T11:01:26.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.684+0000 7fd303338640 1 -- 172.21.15.71:0/3030139765 wait complete. 2024-03-29T11:01:26.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.685+0000 7fd303338640 10 monclient: build_initial_monmap 2024-03-29T11:01:26.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.685+0000 7fd303338640 10 monclient: monmap: 2024-03-29T11:01:26.666 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:01:26.666 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:26.666 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:01:26.666 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:01:26.666 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:26.666 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:26.666 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:01:26.666 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:01:26.666 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:01:26.666 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:26.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.685+0000 7fd303338640 1 librados: starting msgr at 2024-03-29T11:01:26.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.685+0000 7fd303338640 1 librados: starting objecter 2024-03-29T11:01:26.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.686+0000 7fd303338640 1 Processor -- start 2024-03-29T11:01:26.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.686+0000 7fd303338640 1 -- start start 2024-03-29T11:01:26.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.686+0000 7fd303338640 1 librados: setting wanted keys 2024-03-29T11:01:26.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.686+0000 7fd303338640 1 librados: calling monclient init 2024-03-29T11:01:26.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.686+0000 7fd303338640 10 monclient: init 2024-03-29T11:01:26.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.687+0000 7fd303338640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:26.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.687+0000 7fd303338640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:01:26.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.687+0000 7fd303338640 10 monclient(hunting): picked mon.noname-b con 0x7fd2e8004340 addr v1:172.21.15.137:6789/0 2024-03-29T11:01:26.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.687+0000 7fd303338640 10 monclient(hunting): picked mon.noname-c con 0x7fd2e800b300 addr v1:172.21.15.71:6790/0 2024-03-29T11:01:26.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.687+0000 7fd303338640 10 monclient(hunting): picked mon.noname-a con 0x7fd2fc0ac5a0 addr v1:172.21.15.71:6789/0 2024-03-29T11:01:26.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.687+0000 7fd303338640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2fc124130 con 0x7fd2fc0ac5a0 2024-03-29T11:01:26.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.687+0000 7fd303338640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2fc1243a0 con 0x7fd2e8004340 2024-03-29T11:01:26.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.687+0000 7fd303338640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2fc1d0e10 con 0x7fd2e800b300 2024-03-29T11:01:26.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.687+0000 7fd303338640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:26.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.687+0000 7fd303338640 10 monclient(hunting): authenticate will time out at 842.945515s 2024-03-29T11:01:26.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.692+0000 7fd3018d5640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fd2fc0ac5a0 0x7fd2fc1d1d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:50213/0 (socket says 172.21.15.71:50213) 2024-03-29T11:01:26.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.692+0000 7fd3018d5640 1 -- 172.21.15.71:0/1922374402 learned_addr learned my addr 172.21.15.71:0/1922374402 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:01:26.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.692+0000 7fd2f1ffb640 10 client.?.objecter ms_handle_connect 0x7fd2fc0ac5a0 2024-03-29T11:01:26.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.692+0000 7fd2f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:26.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.692+0000 7fd2f1ffb640 10 client.?.objecter ms_handle_connect 0x7fd2e800b300 2024-03-29T11:01:26.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.692+0000 7fd2f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:26.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.692+0000 7fd2f1ffb640 1 -- 172.21.15.71:0/1922374402 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1161622288 0 0) 0x7fd2fc124130 con 0x7fd2fc0ac5a0 2024-03-29T11:01:26.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.692+0000 7fd2f1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:26.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.692+0000 7fd2f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:26.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.692+0000 7fd2f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:26.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.692+0000 7fd2f1ffb640 10 monclient(hunting): my global_id is 4275 2024-03-29T11:01:26.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.692+0000 7fd2f1ffb640 1 -- 172.21.15.71:0/1922374402 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2e0004eb0 con 0x7fd2fc0ac5a0 2024-03-29T11:01:26.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 1 -- 172.21.15.71:0/1922374402 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2967220478 0 0) 0x7fd2fc1d0e10 con 0x7fd2e800b300 2024-03-29T11:01:26.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:26.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:26.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:26.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:01:26.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 1 -- 172.21.15.71:0/1922374402 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2fc124130 con 0x7fd2e800b300 2024-03-29T11:01:26.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 10 client.?.objecter ms_handle_connect 0x7fd2e8004340 2024-03-29T11:01:26.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:26.678 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 1 -- 172.21.15.71:0/1922374402 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1218217213 0 0) 0x7fd2e0004eb0 con 0x7fd2fc0ac5a0 2024-03-29T11:01:26.678 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 1 -- 172.21.15.71:0/1922374402 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2fc1d0e10 con 0x7fd2fc0ac5a0 2024-03-29T11:01:26.678 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 1 -- 172.21.15.71:0/1922374402 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fd2f4001c90 con 0x7fd2fc0ac5a0 2024-03-29T11:01:26.678 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:26.678 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:26.678 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 10 monclient(hunting): dump: 2024-03-29T11:01:26.678 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:26.678 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:26.678 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:26.678 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:26.678 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:26.679 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:26.679 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:26.679 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:26.679 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:26.679 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:26.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:26.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 1 -- 172.21.15.71:0/1922374402 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 756843247 0 0) 0x7fd2fc1243a0 con 0x7fd2e8004340 2024-03-29T11:01:26.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:26.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:26.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:26.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:01:26.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 1 -- 172.21.15.71:0/1922374402 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2e0004eb0 con 0x7fd2e8004340 2024-03-29T11:01:26.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 1 -- 172.21.15.71:0/1922374402 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2638697993 0 0) 0x7fd2fc1d0e10 con 0x7fd2fc0ac5a0 2024-03-29T11:01:26.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 1 -- 172.21.15.71:0/1922374402 >> v1:172.21.15.71:6790/0 conn(0x7fd2e800b300 legacy=0x7fd2fc1cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:26.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 1 -- 172.21.15.71:0/1922374402 >> v1:172.21.15.137:6789/0 conn(0x7fd2e8004340 legacy=0x7fd2fc1d1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:26.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:01:26.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 1 monclient: found mon.a 2024-03-29T11:01:26.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:26.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:01:26.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 1 -- 172.21.15.71:0/1922374402 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2fc144b80 con 0x7fd2fc0ac5a0 2024-03-29T11:01:26.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:01:26.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:01:26.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.696+0000 7fd2f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:26.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.697+0000 7fd2f1ffb640 1 -- 172.21.15.71:0/1922374402 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2f4003990 con 0x7fd2fc0ac5a0 2024-03-29T11:01:26.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.697+0000 7fd2f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:26.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.697+0000 7fd2f1ffb640 1 -- 172.21.15.71:0/1922374402 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fd2f4003da0 con 0x7fd2fc0ac5a0 2024-03-29T11:01:26.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.697+0000 7fd2f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:26.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.697+0000 7fd2f1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:26.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.697+0000 7fd2f1ffb640 10 monclient: dump: 2024-03-29T11:01:26.681 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:26.681 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:26.681 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:26.681 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:26.681 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:26.681 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:26.682 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:26.682 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:26.682 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:26.682 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:26.682 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.697+0000 7fd2f37fe640 4 set_mon_vals no callback set 2024-03-29T11:01:26.683 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.698+0000 7fd303338640 5 monclient: authenticate success, global_id 4275 2024-03-29T11:01:26.683 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.698+0000 7fd303338640 10 monclient: _renew_subs 2024-03-29T11:01:26.683 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.698+0000 7fd303338640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:01:26.683 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.698+0000 7fd303338640 1 -- 172.21.15.71:0/1922374402 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2fc144e30 con 0x7fd2fc0ac5a0 2024-03-29T11:01:26.683 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.698+0000 7fd303338640 10 client.4275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:26.683 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.698+0000 7fd303338640 10 monclient: _renew_subs 2024-03-29T11:01:26.683 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.698+0000 7fd303338640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:01:26.683 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.698+0000 7fd303338640 1 -- 172.21.15.71:0/1922374402 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2fc144030 con 0x7fd2fc0ac5a0 2024-03-29T11:01:26.683 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.698+0000 7fd303338640 1 librados: init done 2024-03-29T11:01:26.683 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.700+0000 7fd2f1ffb640 1 -- 172.21.15.71:0/1922374402 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7fd2f4003f00 con 0x7fd2fc0ac5a0 2024-03-29T11:01:26.683 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.701+0000 7fd303338640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:26.684 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.701+0000 7fd303338640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:26.684 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.701+0000 7fd303338640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:01:26.684 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.701+0000 7fd303338640 1 -- 172.21.15.71:0/1922374402 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2fc075f10 con 0x7fd2fc0ac5a0 2024-03-29T11:01:26.684 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.703+0000 7fd3010d4640 10 monclient: get_auth_request con 0x7fd2e006df10 auth_method 0 2024-03-29T11:01:26.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.705+0000 7fd2f1ffb640 10 client.4275.objecter ms_handle_connect 0x7fd2e006df10 2024-03-29T11:01:26.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.705+0000 7fd2f1ffb640 1 -- 172.21.15.71:0/1922374402 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7fd2f4054c10 con 0x7fd2fc0ac5a0 2024-03-29T11:01:26.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.705+0000 7fd2f1ffb640 10 client.4275.objecter ms_dispatch 0x7fd2fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:26.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.705+0000 7fd2f1ffb640 3 client.4275.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:26.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.705+0000 7fd2f1ffb640 3 client.4275.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:26.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.705+0000 7fd2f1ffb640 20 client.4275.objecter dump_active .. 0 homeless 2024-03-29T11:01:26.692 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.713+0000 7fd2f1ffb640 1 -- 172.21.15.71:0/1922374402 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd2f4050eb0 con 0x7fd2fc0ac5a0 2024-03-29T11:01:26.692 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.713+0000 7fd2f1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:26.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:26.713+0000 7fd2f1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:27.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:27.043+0000 7fd303338640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:27.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:27.043+0000 7fd303338640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:27.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:27.043+0000 7fd303338640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:01:27.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:27.043+0000 7fd303338640 1 -- 172.21.15.71:0/1922374402 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2fc1d0fa0 con 0x7fd2fc0ac5a0 2024-03-29T11:01:27.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:27.044+0000 7fd2f1ffb640 1 -- 172.21.15.71:0/1922374402 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7fd2f4051050 con 0x7fd2fc0ac5a0 2024-03-29T11:01:27.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:27.044+0000 7fd2f1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:27.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:27.044+0000 7fd2f1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:27.029 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:01:27.029 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:27.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:27.051+0000 7fd303338640 10 librados: watch_flush enter 2024-03-29T11:01:27.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:27.052+0000 7fd303338640 10 librados: watch_flush exit 2024-03-29T11:01:27.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:27.052+0000 7fd303338640 20 client.4275.objecter shutdown clearing up homeless session... 2024-03-29T11:01:27.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:27.052+0000 7fd303338640 10 client.4275.objecter successfully canceled tick 2024-03-29T11:01:27.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:27.052+0000 7fd303338640 1 -- 172.21.15.71:0/1922374402 >> v1:172.21.15.137:6816/973332665 conn(0x7fd2e006df10 legacy=0x7fd2e00703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:27.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:27.052+0000 7fd303338640 10 monclient: shutdown 2024-03-29T11:01:27.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:27.052+0000 7fd303338640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:27.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:27.052+0000 7fd303338640 1 -- 172.21.15.71:0/1922374402 >> v1:172.21.15.71:6789/0 conn(0x7fd2fc0ac5a0 legacy=0x7fd2fc1d1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:27.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:27.053+0000 7fd303338640 1 -- 172.21.15.71:0/1922374402 shutdown_connections 2024-03-29T11:01:27.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:27.053+0000 7fd303338640 1 -- 172.21.15.71:0/1922374402 >> 172.21.15.71:0/1922374402 conn(0x7fd2fc06b660 msgr2=0x7fd2fc0a9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:27.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:27.053+0000 7fd303338640 1 -- 172.21.15.71:0/1922374402 shutdown_connections 2024-03-29T11:01:27.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:27.053+0000 7fd303338640 1 -- 172.21.15.71:0/1922374402 wait complete. 2024-03-29T11:01:27.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:27.053+0000 7fd303338640 1 librados: shutdown 2024-03-29T11:01:27.049 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:33.857 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:34.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.026+0000 7f9a69486640 10 monclient: get_monmap_and_config 2024-03-29T11:01:34.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.026+0000 7f9a69486640 10 monclient: build_initial_monmap 2024-03-29T11:01:34.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.026+0000 7f9a69486640 10 monclient: monmap: 2024-03-29T11:01:34.009 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:01:34.009 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:34.009 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:01:34.028431+0000 2024-03-29T11:01:34.009 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:01:34.028431+0000 2024-03-29T11:01:34.009 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:34.009 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:34.009 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:01:34.009 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:01:34.009 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:01:34.009 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:34.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.028+0000 7f9a69486640 1 Processor -- start 2024-03-29T11:01:34.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.028+0000 7f9a69486640 1 -- start start 2024-03-29T11:01:34.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.028+0000 7f9a69486640 10 monclient: init 2024-03-29T11:01:34.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.029+0000 7f9a69486640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:34.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.029+0000 7f9a69486640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:01:34.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.029+0000 7f9a69486640 10 monclient(hunting): picked mon.noname-c con 0x7f9a640b4100 addr v1:172.21.15.137:6789/0 2024-03-29T11:01:34.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.036+0000 7f9a69486640 10 monclient(hunting): picked mon.noname-a con 0x7f9a6413ee90 addr v1:172.21.15.71:6789/0 2024-03-29T11:01:34.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.036+0000 7f9a69486640 10 monclient(hunting): picked mon.noname-b con 0x7f9a640af900 addr v1:172.21.15.71:6790/0 2024-03-29T11:01:34.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.036+0000 7f9a69486640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a640afd10 con 0x7f9a6413ee90 2024-03-29T11:01:34.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.036+0000 7f9a69486640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a640af6f0 con 0x7f9a640b4100 2024-03-29T11:01:34.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.036+0000 7f9a69486640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a640aac80 con 0x7f9a640af900 2024-03-29T11:01:34.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.036+0000 7f9a69486640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:34.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.036+0000 7f9a69486640 10 monclient(hunting): authenticate will time out at 850.295301s 2024-03-29T11:01:34.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.036+0000 7f9a627fc640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f9a6413ee90 0x7f9a6413b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:34911/0 (socket says 172.21.15.71:34911) 2024-03-29T11:01:34.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.036+0000 7f9a627fc640 1 -- 172.21.15.71:0/111780753 learned_addr learned my addr 172.21.15.71:0/111780753 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:01:34.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.044+0000 7f9a61ffb640 1 -- 172.21.15.71:0/111780753 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174560686 0 0) 0x7f9a640afd10 con 0x7f9a6413ee90 2024-03-29T11:01:34.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.044+0000 7f9a61ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:34.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.044+0000 7f9a61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:34.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.044+0000 7f9a61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:34.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.044+0000 7f9a61ffb640 10 monclient(hunting): my global_id is 4278 2024-03-29T11:01:34.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.044+0000 7f9a61ffb640 1 -- 172.21.15.71:0/111780753 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a4c003520 con 0x7f9a6413ee90 2024-03-29T11:01:34.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.045+0000 7f9a61ffb640 1 -- 172.21.15.71:0/111780753 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2501303859 0 0) 0x7f9a640af6f0 con 0x7f9a640b4100 2024-03-29T11:01:34.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.045+0000 7f9a61ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:34.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.045+0000 7f9a61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:34.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.045+0000 7f9a61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:34.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.045+0000 7f9a61ffb640 10 monclient(hunting): my global_id is 4252 2024-03-29T11:01:34.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.045+0000 7f9a61ffb640 1 -- 172.21.15.71:0/111780753 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a640afd10 con 0x7f9a640b4100 2024-03-29T11:01:34.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.045+0000 7f9a61ffb640 1 -- 172.21.15.71:0/111780753 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 302856316 0 0) 0x7f9a4c003520 con 0x7f9a6413ee90 2024-03-29T11:01:34.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.045+0000 7f9a61ffb640 1 -- 172.21.15.71:0/111780753 >> v1:172.21.15.71:6790/0 conn(0x7f9a640af900 legacy=0x7f9a6414be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:34.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.046+0000 7f9a61ffb640 1 -- 172.21.15.71:0/111780753 >> v1:172.21.15.137:6789/0 conn(0x7f9a640b4100 legacy=0x7f9a640adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:34.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.046+0000 7f9a61ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:01:34.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.046+0000 7f9a61ffb640 1 monclient: found mon.noname-a 2024-03-29T11:01:34.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.046+0000 7f9a61ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:34.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.046+0000 7f9a61ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:01:34.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.046+0000 7f9a61ffb640 1 -- 172.21.15.71:0/111780753 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a640aaf60 con 0x7f9a6413ee90 2024-03-29T11:01:34.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.046+0000 7f9a61ffb640 10 monclient: _finish_auth 0 2024-03-29T11:01:34.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.046+0000 7f9a61ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:01:34.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.046+0000 7f9a61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:34.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.046+0000 7f9a61ffb640 1 -- 172.21.15.71:0/111780753 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9a58001610 con 0x7f9a6413ee90 2024-03-29T11:01:34.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.046+0000 7f9a61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:34.029 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.046+0000 7f9a61ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:34.029 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.046+0000 7f9a61ffb640 10 monclient: dump: 2024-03-29T11:01:34.029 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:34.029 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:34.029 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:34.029 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:34.029 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:34.029 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:34.029 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:34.029 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:34.029 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:34.029 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:34.029 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.046+0000 7f9a69486640 5 monclient: authenticate success, global_id 4278 2024-03-29T11:01:34.029 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.046+0000 7f9a69486640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:34.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.046+0000 7f9a61ffb640 1 -- 172.21.15.71:0/111780753 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a58001770 con 0x7f9a6413ee90 2024-03-29T11:01:34.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.046+0000 7f9a61ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:34.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.046+0000 7f9a61ffb640 1 -- 172.21.15.71:0/111780753 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9a58001b80 con 0x7f9a6413ee90 2024-03-29T11:01:34.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.046+0000 7f9a61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:34.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.046+0000 7f9a61ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:34.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.046+0000 7f9a61ffb640 10 monclient: dump: 2024-03-29T11:01:34.030 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:34.030 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:34.030 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:34.030 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:34.030 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:34.030 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:34.030 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:34.030 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:34.031 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:34.031 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:34.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.046+0000 7f9a69486640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:34.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.046+0000 7f9a69486640 4 set_mon_vals no callback set 2024-03-29T11:01:34.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.048+0000 7f9a69486640 10 monclient: shutdown 2024-03-29T11:01:34.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.048+0000 7f9a69486640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:34.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.048+0000 7f9a69486640 1 -- 172.21.15.71:0/111780753 >> v1:172.21.15.71:6789/0 conn(0x7f9a6413ee90 legacy=0x7f9a6413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:34.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.050+0000 7f9a69486640 1 -- 172.21.15.71:0/111780753 shutdown_connections 2024-03-29T11:01:34.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.050+0000 7f9a69486640 1 -- 172.21.15.71:0/111780753 >> 172.21.15.71:0/111780753 conn(0x7f9a640b35e0 msgr2=0x7f9a640b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:34.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.051+0000 7f9a69486640 1 -- 172.21.15.71:0/111780753 shutdown_connections 2024-03-29T11:01:34.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.051+0000 7f9a69486640 1 -- 172.21.15.71:0/111780753 wait complete. 2024-03-29T11:01:34.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.052+0000 7f9a69486640 10 monclient: build_initial_monmap 2024-03-29T11:01:34.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.052+0000 7f9a69486640 10 monclient: monmap: 2024-03-29T11:01:34.035 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:01:34.035 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:34.035 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:01:34.035 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:01:34.035 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:34.035 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:34.035 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:01:34.035 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:01:34.035 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:01:34.035 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:34.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.053+0000 7f9a69486640 1 librados: starting msgr at 2024-03-29T11:01:34.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.053+0000 7f9a69486640 1 librados: starting objecter 2024-03-29T11:01:34.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.053+0000 7f9a69486640 1 Processor -- start 2024-03-29T11:01:34.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.053+0000 7f9a69486640 1 -- start start 2024-03-29T11:01:34.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.053+0000 7f9a69486640 1 librados: setting wanted keys 2024-03-29T11:01:34.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.053+0000 7f9a69486640 1 librados: calling monclient init 2024-03-29T11:01:34.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.053+0000 7f9a69486640 10 monclient: init 2024-03-29T11:01:34.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.054+0000 7f9a69486640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:34.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.054+0000 7f9a69486640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:01:34.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.054+0000 7f9a69486640 10 monclient(hunting): picked mon.noname-a con 0x7f9a640af900 addr v1:172.21.15.71:6789/0 2024-03-29T11:01:34.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.056+0000 7f9a69486640 10 monclient(hunting): picked mon.noname-b con 0x7f9a640b4100 addr v1:172.21.15.137:6789/0 2024-03-29T11:01:34.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.056+0000 7f9a69486640 10 monclient(hunting): picked mon.noname-c con 0x7f9a6413ee90 addr v1:172.21.15.71:6790/0 2024-03-29T11:01:34.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.056+0000 7f9a69486640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a641407b0 con 0x7f9a640af900 2024-03-29T11:01:34.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.056+0000 7f9a69486640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a64071fe0 con 0x7f9a640b4100 2024-03-29T11:01:34.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.056+0000 7f9a69486640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a6413a930 con 0x7f9a6413ee90 2024-03-29T11:01:34.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.056+0000 7f9a69486640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:34.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.056+0000 7f9a69486640 10 monclient(hunting): authenticate will time out at 850.315858s 2024-03-29T11:01:34.040 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.059+0000 7f9a62ffd640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f9a640af900 0x7f9a6413a220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:35069/0 (socket says 172.21.15.71:35069) 2024-03-29T11:01:34.040 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.059+0000 7f9a62ffd640 1 -- 172.21.15.71:0/183224447 learned_addr learned my addr 172.21.15.71:0/183224447 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:01:34.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.059+0000 7f9a4b7fe640 10 client.?.objecter ms_handle_connect 0x7f9a640af900 2024-03-29T11:01:34.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:34.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 10 client.?.objecter ms_handle_connect 0x7f9a640b4100 2024-03-29T11:01:34.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:34.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 1 -- 172.21.15.71:0/183224447 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512153106 0 0) 0x7f9a641407b0 con 0x7f9a640af900 2024-03-29T11:01:34.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:01:34.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:34.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:34.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 10 monclient(hunting): my global_id is 4281 2024-03-29T11:01:34.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 1 -- 172.21.15.71:0/183224447 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a50004eb0 con 0x7f9a640af900 2024-03-29T11:01:34.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 1 -- 172.21.15.71:0/183224447 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2423731437 0 0) 0x7f9a64071fe0 con 0x7f9a640b4100 2024-03-29T11:01:34.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:01:34.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:34.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:34.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 10 monclient(hunting): my global_id is 4255 2024-03-29T11:01:34.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 1 -- 172.21.15.71:0/183224447 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a50006230 con 0x7f9a640b4100 2024-03-29T11:01:34.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 1 -- 172.21.15.71:0/183224447 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3265590167 0 0) 0x7f9a50004eb0 con 0x7f9a640af900 2024-03-29T11:01:34.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 1 -- 172.21.15.71:0/183224447 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a500066a0 con 0x7f9a640af900 2024-03-29T11:01:34.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 1 -- 172.21.15.71:0/183224447 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9a54001980 con 0x7f9a640af900 2024-03-29T11:01:34.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:34.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:34.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 10 monclient(hunting): dump: 2024-03-29T11:01:34.045 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:34.045 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:34.045 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:34.045 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:34.045 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:34.045 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:34.045 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:34.045 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:34.045 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:34.045 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:34.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:34.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 1 -- 172.21.15.71:0/183224447 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 516627103 0 0) 0x7f9a50006230 con 0x7f9a640b4100 2024-03-29T11:01:34.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 1 -- 172.21.15.71:0/183224447 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a50006d90 con 0x7f9a640b4100 2024-03-29T11:01:34.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 1 -- 172.21.15.71:0/183224447 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9a58001b40 con 0x7f9a640b4100 2024-03-29T11:01:34.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:34.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:34.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 10 monclient(hunting): dump: 2024-03-29T11:01:34.046 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:34.046 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:34.046 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:34.046 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:34.046 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:34.046 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:34.046 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:34.046 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:34.046 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:34.046 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:34.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 1 -- 172.21.15.71:0/183224447 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1771443910 0 0) 0x7f9a500066a0 con 0x7f9a640af900 2024-03-29T11:01:34.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.060+0000 7f9a4b7fe640 1 -- 172.21.15.71:0/183224447 >> v1:172.21.15.71:6790/0 conn(0x7f9a6413ee90 legacy=0x7f9a641e6d30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:01:34.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.061+0000 7f9a4b7fe640 1 -- 172.21.15.71:0/183224447 >> v1:172.21.15.137:6789/0 conn(0x7f9a640b4100 legacy=0x7f9a641db4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:34.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.061+0000 7f9a4b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:01:34.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.061+0000 7f9a4b7fe640 1 monclient: found mon.a 2024-03-29T11:01:34.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.061+0000 7f9a4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:34.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.061+0000 7f9a4b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:01:34.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.061+0000 7f9a4b7fe640 1 -- 172.21.15.71:0/183224447 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6413f5c0 con 0x7f9a640af900 2024-03-29T11:01:34.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.061+0000 7f9a4b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:01:34.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.061+0000 7f9a4b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:01:34.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.061+0000 7f9a4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:34.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.061+0000 7f9a69486640 5 monclient: authenticate success, global_id 4281 2024-03-29T11:01:34.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.061+0000 7f9a69486640 10 monclient: _renew_subs 2024-03-29T11:01:34.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.061+0000 7f9a69486640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:01:34.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.061+0000 7f9a69486640 1 -- 172.21.15.71:0/183224447 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a6413f810 con 0x7f9a640af900 2024-03-29T11:01:34.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.061+0000 7f9a69486640 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:34.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.061+0000 7f9a69486640 10 monclient: _renew_subs 2024-03-29T11:01:34.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.061+0000 7f9a69486640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:01:34.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.061+0000 7f9a69486640 1 -- 172.21.15.71:0/183224447 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a64143250 con 0x7f9a640af900 2024-03-29T11:01:34.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.061+0000 7f9a69486640 1 librados: init done 2024-03-29T11:01:34.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.062+0000 7f9a4b7fe640 1 -- 172.21.15.71:0/183224447 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a540035f0 con 0x7f9a640af900 2024-03-29T11:01:34.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.062+0000 7f9a4b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:34.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.062+0000 7f9a4b7fe640 1 -- 172.21.15.71:0/183224447 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9a54001980 con 0x7f9a640af900 2024-03-29T11:01:34.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.062+0000 7f9a4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:34.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.062+0000 7f9a4b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:34.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.062+0000 7f9a4b7fe640 10 monclient: dump: 2024-03-29T11:01:34.050 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:34.050 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:34.050 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:34.050 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:34.050 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:34.050 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:34.050 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:34.050 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:34.050 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:34.050 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:34.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.062+0000 7f9a60ff9640 4 set_mon_vals no callback set 2024-03-29T11:01:34.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.064+0000 7f9a4b7fe640 1 -- 172.21.15.71:0/183224447 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f9a54003860 con 0x7f9a640af900 2024-03-29T11:01:34.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.065+0000 7f9a627fc640 10 monclient: get_auth_request con 0x7f9a5006ea60 auth_method 0 2024-03-29T11:01:34.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.065+0000 7f9a4b7fe640 1 -- 172.21.15.71:0/183224447 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f9a540543d0 con 0x7f9a640af900 2024-03-29T11:01:34.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.065+0000 7f9a4b7fe640 10 client.4281.objecter ms_dispatch 0x7f9a64000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:34.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.065+0000 7f9a4b7fe640 3 client.4281.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:34.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.065+0000 7f9a4b7fe640 3 client.4281.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:34.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.065+0000 7f9a4b7fe640 20 client.4281.objecter dump_active .. 0 homeless 2024-03-29T11:01:34.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.065+0000 7f9a4b7fe640 10 client.4281.objecter ms_handle_connect 0x7f9a5006ea60 2024-03-29T11:01:34.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.066+0000 7f9a69486640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:34.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.066+0000 7f9a69486640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:34.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.066+0000 7f9a69486640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:01:34.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.066+0000 7f9a69486640 1 -- 172.21.15.71:0/183224447 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a44005080 con 0x7f9a640af900 2024-03-29T11:01:34.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.077+0000 7f9a4b7fe640 1 -- 172.21.15.71:0/183224447 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9a54050670 con 0x7f9a640af900 2024-03-29T11:01:34.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.077+0000 7f9a4b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:34.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.077+0000 7f9a4b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:34.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.402+0000 7f9a69486640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:34.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.402+0000 7f9a69486640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:34.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.403+0000 7f9a69486640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:01:34.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.403+0000 7f9a69486640 1 -- 172.21.15.71:0/183224447 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a44005330 con 0x7f9a640af900 2024-03-29T11:01:34.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.405+0000 7f9a4b7fe640 1 -- 172.21.15.71:0/183224447 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f9a54050670 con 0x7f9a640af900 2024-03-29T11:01:34.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.405+0000 7f9a4b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:34.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.405+0000 7f9a4b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:34.387 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:01:34.387 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:34.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.410+0000 7f9a69486640 10 librados: watch_flush enter 2024-03-29T11:01:34.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.410+0000 7f9a69486640 10 librados: watch_flush exit 2024-03-29T11:01:34.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.410+0000 7f9a69486640 20 client.4281.objecter shutdown clearing up homeless session... 2024-03-29T11:01:34.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.410+0000 7f9a69486640 10 client.4281.objecter successfully canceled tick 2024-03-29T11:01:34.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.414+0000 7f9a69486640 1 -- 172.21.15.71:0/183224447 >> v1:172.21.15.137:6816/973332665 conn(0x7f9a5006ea60 legacy=0x7f9a50070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:34.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.414+0000 7f9a69486640 10 monclient: shutdown 2024-03-29T11:01:34.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.414+0000 7f9a69486640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:34.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.414+0000 7f9a69486640 1 -- 172.21.15.71:0/183224447 >> v1:172.21.15.71:6789/0 conn(0x7f9a640af900 legacy=0x7f9a6413a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:34.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.414+0000 7f9a69486640 1 -- 172.21.15.71:0/183224447 shutdown_connections 2024-03-29T11:01:34.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.414+0000 7f9a69486640 1 -- 172.21.15.71:0/183224447 >> 172.21.15.71:0/183224447 conn(0x7f9a640b35e0 msgr2=0x7f9a641d1570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:34.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.415+0000 7f9a69486640 1 -- 172.21.15.71:0/183224447 shutdown_connections 2024-03-29T11:01:34.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.415+0000 7f9a69486640 1 -- 172.21.15.71:0/183224447 wait complete. 2024-03-29T11:01:34.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:34.415+0000 7f9a69486640 1 librados: shutdown 2024-03-29T11:01:34.412 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:41.219 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:41.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.398+0000 7f7f3ee43640 10 monclient: get_monmap_and_config 2024-03-29T11:01:41.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.398+0000 7f7f3ee43640 10 monclient: build_initial_monmap 2024-03-29T11:01:41.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.398+0000 7f7f3ee43640 10 monclient: monmap: 2024-03-29T11:01:41.380 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:01:41.380 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:41.380 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:01:41.400282+0000 2024-03-29T11:01:41.380 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:01:41.400282+0000 2024-03-29T11:01:41.380 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:41.380 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:41.380 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:01:41.380 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:01:41.381 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:01:41.381 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:41.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.403+0000 7f7f3ee43640 1 Processor -- start 2024-03-29T11:01:41.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.403+0000 7f7f3ee43640 1 -- start start 2024-03-29T11:01:41.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.403+0000 7f7f3ee43640 10 monclient: init 2024-03-29T11:01:41.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.404+0000 7f7f3ee43640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:41.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.404+0000 7f7f3ee43640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:01:41.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.404+0000 7f7f3ee43640 10 monclient(hunting): picked mon.noname-b con 0x7f7f380a9ac0 addr v1:172.21.15.71:6790/0 2024-03-29T11:01:41.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.407+0000 7f7f3ee43640 10 monclient(hunting): picked mon.noname-c con 0x7f7f3813ee90 addr v1:172.21.15.137:6789/0 2024-03-29T11:01:41.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.407+0000 7f7f3cbdf640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f7f380a9ac0 0x7f7f380adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:40915/0 (socket says 172.21.15.71:40915) 2024-03-29T11:01:41.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.408+0000 7f7f3cbdf640 1 -- 172.21.15.71:0/2895944922 learned_addr learned my addr 172.21.15.71:0/2895944922 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:01:41.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.408+0000 7f7f3ee43640 10 monclient(hunting): picked mon.noname-a con 0x7f7f380af900 addr v1:172.21.15.71:6789/0 2024-03-29T11:01:41.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.408+0000 7f7f3ee43640 1 -- 172.21.15.71:0/2895944922 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f380afd10 con 0x7f7f380af900 2024-03-29T11:01:41.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.408+0000 7f7f3ee43640 1 -- 172.21.15.71:0/2895944922 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f380af6f0 con 0x7f7f3813ee90 2024-03-29T11:01:41.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.408+0000 7f7f3ee43640 1 -- 172.21.15.71:0/2895944922 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f380aac80 con 0x7f7f380a9ac0 2024-03-29T11:01:41.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.408+0000 7f7f3ee43640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:41.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.408+0000 7f7f3ee43640 10 monclient(hunting): authenticate will time out at 857.667310s 2024-03-29T11:01:41.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.413+0000 7f7f3cbdf640 1 -- 172.21.15.71:0/2895944922 >> v1:172.21.15.71:6790/0 conn(0x7f7f380a9ac0 legacy=0x7f7f380adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-29T11:01:41.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.413+0000 7f7f3cbdf640 1 -- 172.21.15.71:0/2895944922 >> v1:172.21.15.71:6790/0 conn(0x7f7f380a9ac0 legacy=0x7f7f380adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:01:41.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.413+0000 7f7f3cbdf640 1 --1- 172.21.15.71:0/2895944922 >> v1:172.21.15.71:6790/0 conn(0x7f7f380a9ac0 0x7f7f380adfe0 :-1 s=OPENED pgs=92913 cs=1 l=1).handle_message read tag failed 2024-03-29T11:01:41.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.413+0000 7f7f3cbdf640 1 --1- 172.21.15.71:0/2895944922 >> v1:172.21.15.71:6790/0 conn(0x7f7f380a9ac0 0x7f7f380adfe0 :-1 s=OPENED pgs=92913 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:01:41.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.413+0000 7f7f377fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.71:6790/0 2024-03-29T11:01:41.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.413+0000 7f7f377fe640 1 -- 172.21.15.71:0/2895944922 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713399526 0 0) 0x7f7f380afd10 con 0x7f7f380af900 2024-03-29T11:01:41.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.413+0000 7f7f377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:01:41.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.413+0000 7f7f377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:41.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.413+0000 7f7f377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:41.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.413+0000 7f7f377fe640 10 monclient(hunting): my global_id is 4284 2024-03-29T11:01:41.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.413+0000 7f7f377fe640 1 -- 172.21.15.71:0/2895944922 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f28003520 con 0x7f7f380af900 2024-03-29T11:01:41.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.414+0000 7f7f377fe640 1 -- 172.21.15.71:0/2895944922 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371566238 0 0) 0x7f7f380af6f0 con 0x7f7f3813ee90 2024-03-29T11:01:41.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.414+0000 7f7f377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:01:41.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.414+0000 7f7f377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:41.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.414+0000 7f7f377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:41.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.414+0000 7f7f377fe640 10 monclient(hunting): my global_id is 4258 2024-03-29T11:01:41.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.414+0000 7f7f377fe640 1 -- 172.21.15.71:0/2895944922 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f380afd10 con 0x7f7f3813ee90 2024-03-29T11:01:41.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.414+0000 7f7f377fe640 1 -- 172.21.15.71:0/2895944922 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1844086401 0 0) 0x7f7f28003520 con 0x7f7f380af900 2024-03-29T11:01:41.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.414+0000 7f7f377fe640 1 -- 172.21.15.71:0/2895944922 >> v1:172.21.15.71:6790/0 conn(0x7f7f380a9ac0 legacy=0x7f7f380adfe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:01:41.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.414+0000 7f7f377fe640 1 -- 172.21.15.71:0/2895944922 >> v1:172.21.15.137:6789/0 conn(0x7f7f3813ee90 legacy=0x7f7f3813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:41.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.414+0000 7f7f377fe640 10 monclient: _finish_hunting 0 2024-03-29T11:01:41.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.414+0000 7f7f377fe640 1 monclient: found mon.noname-a 2024-03-29T11:01:41.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.414+0000 7f7f377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:41.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.414+0000 7f7f377fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:01:41.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.414+0000 7f7f377fe640 1 -- 172.21.15.71:0/2895944922 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f380aaf60 con 0x7f7f380af900 2024-03-29T11:01:41.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.415+0000 7f7f377fe640 10 monclient: _finish_auth 0 2024-03-29T11:01:41.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.415+0000 7f7f377fe640 10 monclient: _check_auth_tickets 2024-03-29T11:01:41.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.415+0000 7f7f377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:41.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.415+0000 7f7f377fe640 1 -- 172.21.15.71:0/2895944922 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f7f30001a10 con 0x7f7f380af900 2024-03-29T11:01:41.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.415+0000 7f7f377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:41.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.415+0000 7f7f377fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:41.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.415+0000 7f7f377fe640 10 monclient: dump: 2024-03-29T11:01:41.397 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:41.397 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:41.397 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:41.398 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:41.398 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:41.398 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:41.398 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:41.398 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:41.398 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:41.398 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:41.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.415+0000 7f7f3ee43640 5 monclient: authenticate success, global_id 4284 2024-03-29T11:01:41.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.415+0000 7f7f3ee43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:41.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.415+0000 7f7f377fe640 1 -- 172.21.15.71:0/2895944922 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f30001c30 con 0x7f7f380af900 2024-03-29T11:01:41.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.415+0000 7f7f377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:41.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.415+0000 7f7f377fe640 1 -- 172.21.15.71:0/2895944922 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f7f30001fc0 con 0x7f7f380af900 2024-03-29T11:01:41.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.415+0000 7f7f377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:41.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.415+0000 7f7f377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:41.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.415+0000 7f7f377fe640 10 monclient: dump: 2024-03-29T11:01:41.399 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:41.399 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:41.399 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:41.399 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:41.399 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:41.399 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:41.399 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:41.399 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:41.399 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:41.399 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:41.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.415+0000 7f7f3ee43640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:41.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.415+0000 7f7f3ee43640 4 set_mon_vals no callback set 2024-03-29T11:01:41.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.416+0000 7f7f3ee43640 10 monclient: shutdown 2024-03-29T11:01:41.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.416+0000 7f7f3ee43640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:41.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.416+0000 7f7f3ee43640 1 -- 172.21.15.71:0/2895944922 >> v1:172.21.15.71:6789/0 conn(0x7f7f380af900 legacy=0x7f7f38149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:41.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.417+0000 7f7f3ee43640 1 -- 172.21.15.71:0/2895944922 shutdown_connections 2024-03-29T11:01:41.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.417+0000 7f7f3ee43640 1 -- 172.21.15.71:0/2895944922 >> 172.21.15.71:0/2895944922 conn(0x7f7f380b35e0 msgr2=0x7f7f380b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:41.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.417+0000 7f7f3ee43640 1 -- 172.21.15.71:0/2895944922 shutdown_connections 2024-03-29T11:01:41.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.417+0000 7f7f3ee43640 1 -- 172.21.15.71:0/2895944922 wait complete. 2024-03-29T11:01:41.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.418+0000 7f7f3ee43640 10 monclient: build_initial_monmap 2024-03-29T11:01:41.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.418+0000 7f7f3ee43640 10 monclient: monmap: 2024-03-29T11:01:41.402 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:01:41.402 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:41.402 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:01:41.402 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:01:41.402 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:41.403 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:41.403 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:01:41.403 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:01:41.403 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:01:41.403 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:41.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.418+0000 7f7f3ee43640 1 librados: starting msgr at 2024-03-29T11:01:41.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.418+0000 7f7f3ee43640 1 librados: starting objecter 2024-03-29T11:01:41.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.419+0000 7f7f3ee43640 1 Processor -- start 2024-03-29T11:01:41.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.419+0000 7f7f3ee43640 1 -- start start 2024-03-29T11:01:41.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.419+0000 7f7f3ee43640 1 librados: setting wanted keys 2024-03-29T11:01:41.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.419+0000 7f7f3ee43640 1 librados: calling monclient init 2024-03-29T11:01:41.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.419+0000 7f7f3ee43640 10 monclient: init 2024-03-29T11:01:41.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.419+0000 7f7f3ee43640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:41.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.419+0000 7f7f3ee43640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:01:41.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.419+0000 7f7f3ee43640 10 monclient(hunting): picked mon.noname-c con 0x7f7f380a9ac0 addr v1:172.21.15.71:6790/0 2024-03-29T11:01:41.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.422+0000 7f7f3ee43640 10 monclient(hunting): picked mon.noname-b con 0x7f7f380af900 addr v1:172.21.15.137:6789/0 2024-03-29T11:01:41.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.422+0000 7f7f3ee43640 10 monclient(hunting): picked mon.noname-a con 0x7f7f3813ee90 addr v1:172.21.15.71:6789/0 2024-03-29T11:01:41.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.422+0000 7f7f3ee43640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f38134b90 con 0x7f7f3813ee90 2024-03-29T11:01:41.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.422+0000 7f7f3ee43640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f381331a0 con 0x7f7f380af900 2024-03-29T11:01:41.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.422+0000 7f7f3ee43640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f38134e00 con 0x7f7f380a9ac0 2024-03-29T11:01:41.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.422+0000 7f7f3ee43640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:41.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.422+0000 7f7f3ee43640 10 monclient(hunting): authenticate will time out at 857.681474s 2024-03-29T11:01:41.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.427+0000 7f7f37fff640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f7f380af900 0x7f7f38071610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:59085/0 (socket says 172.21.15.71:59085) 2024-03-29T11:01:41.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.427+0000 7f7f37fff640 1 -- 172.21.15.71:0/1718483667 learned_addr learned my addr 172.21.15.71:0/1718483667 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:01:41.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.428+0000 7f7f357fa640 10 client.?.objecter ms_handle_connect 0x7f7f380af900 2024-03-29T11:01:41.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.428+0000 7f7f357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:41.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.428+0000 7f7f357fa640 10 client.?.objecter ms_handle_connect 0x7f7f3813ee90 2024-03-29T11:01:41.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.428+0000 7f7f357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:41.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.428+0000 7f7f357fa640 1 -- 172.21.15.71:0/1718483667 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193947589 0 0) 0x7f7f381331a0 con 0x7f7f380af900 2024-03-29T11:01:41.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.428+0000 7f7f357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:01:41.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.428+0000 7f7f357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:41.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.428+0000 7f7f357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:41.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.428+0000 7f7f357fa640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:01:41.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.428+0000 7f7f357fa640 1 -- 172.21.15.71:0/1718483667 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f1c004eb0 con 0x7f7f380af900 2024-03-29T11:01:41.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.428+0000 7f7f357fa640 1 -- 172.21.15.71:0/1718483667 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1341749996 0 0) 0x7f7f38134b90 con 0x7f7f3813ee90 2024-03-29T11:01:41.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.428+0000 7f7f357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:01:41.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.428+0000 7f7f357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:41.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.428+0000 7f7f357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:41.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.428+0000 7f7f357fa640 10 monclient(hunting): my global_id is 4287 2024-03-29T11:01:41.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.428+0000 7f7f357fa640 1 -- 172.21.15.71:0/1718483667 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f381331a0 con 0x7f7f3813ee90 2024-03-29T11:01:41.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.429+0000 7f7f357fa640 1 -- 172.21.15.71:0/1718483667 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3641905992 0 0) 0x7f7f1c004eb0 con 0x7f7f380af900 2024-03-29T11:01:41.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.429+0000 7f7f357fa640 1 -- 172.21.15.71:0/1718483667 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f38134b90 con 0x7f7f380af900 2024-03-29T11:01:41.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.429+0000 7f7f357fa640 1 -- 172.21.15.71:0/1718483667 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f7f24001990 con 0x7f7f380af900 2024-03-29T11:01:41.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.429+0000 7f7f357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:41.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.429+0000 7f7f357fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:41.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.429+0000 7f7f357fa640 10 monclient(hunting): dump: 2024-03-29T11:01:41.412 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:41.412 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:41.412 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:41.412 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:41.412 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:41.412 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:41.412 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:41.412 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:41.412 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:41.412 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:41.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.429+0000 7f7f357fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:41.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.432+0000 7f7f357fa640 1 -- 172.21.15.71:0/1718483667 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3030999626 0 0) 0x7f7f38134b90 con 0x7f7f380af900 2024-03-29T11:01:41.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.432+0000 7f7f357fa640 1 -- 172.21.15.71:0/1718483667 >> v1:172.21.15.71:6790/0 conn(0x7f7f380a9ac0 legacy=0x7f7f38132a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:41.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.432+0000 7f7f357fa640 1 -- 172.21.15.71:0/1718483667 >> v1:172.21.15.71:6789/0 conn(0x7f7f3813ee90 legacy=0x7f7f3812d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:41.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.432+0000 7f7f357fa640 10 monclient: _finish_hunting 0 2024-03-29T11:01:41.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.432+0000 7f7f357fa640 1 monclient: found mon.b 2024-03-29T11:01:41.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.432+0000 7f7f357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:41.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.432+0000 7f7f357fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:01:41.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.432+0000 7f7f357fa640 1 -- 172.21.15.71:0/1718483667 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f381350e0 con 0x7f7f380af900 2024-03-29T11:01:41.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.432+0000 7f7f357fa640 10 monclient: _finish_auth 0 2024-03-29T11:01:41.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.432+0000 7f7f357fa640 10 monclient: _check_auth_tickets 2024-03-29T11:01:41.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.432+0000 7f7f357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:41.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.432+0000 7f7f357fa640 10 client.?.objecter ms_handle_connect 0x7f7f380a9ac0 2024-03-29T11:01:41.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.432+0000 7f7f357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:41.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.432+0000 7f7f3ee43640 5 monclient: authenticate success, global_id 4261 2024-03-29T11:01:41.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.432+0000 7f7f3ee43640 10 monclient: _renew_subs 2024-03-29T11:01:41.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.432+0000 7f7f3ee43640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:01:41.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.432+0000 7f7f3ee43640 1 -- 172.21.15.71:0/1718483667 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f38072190 con 0x7f7f380af900 2024-03-29T11:01:41.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.432+0000 7f7f3ee43640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:41.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.432+0000 7f7f3ee43640 10 monclient: _renew_subs 2024-03-29T11:01:41.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.432+0000 7f7f3ee43640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:01:41.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.432+0000 7f7f3ee43640 1 -- 172.21.15.71:0/1718483667 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f38071f00 con 0x7f7f380af900 2024-03-29T11:01:41.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.432+0000 7f7f3ee43640 1 librados: init done 2024-03-29T11:01:41.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.433+0000 7f7f357fa640 1 -- 172.21.15.71:0/1718483667 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f240035f0 con 0x7f7f380af900 2024-03-29T11:01:41.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.433+0000 7f7f357fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:41.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.433+0000 7f7f367fc640 4 set_mon_vals no callback set 2024-03-29T11:01:41.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.434+0000 7f7f357fa640 1 -- 172.21.15.71:0/1718483667 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f7f240039c0 con 0x7f7f380af900 2024-03-29T11:01:41.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.434+0000 7f7f357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:41.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.434+0000 7f7f357fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:41.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.434+0000 7f7f357fa640 10 monclient: dump: 2024-03-29T11:01:41.419 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:41.419 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:41.419 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:41.419 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:41.419 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:41.419 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:41.419 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:41.419 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:41.419 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:41.419 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:41.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.435+0000 7f7f357fa640 1 -- 172.21.15.71:0/1718483667 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f7f24003be0 con 0x7f7f380af900 2024-03-29T11:01:41.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.436+0000 7f7f357fa640 1 -- 172.21.15.71:0/1718483667 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f7f24054740 con 0x7f7f380af900 2024-03-29T11:01:41.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.436+0000 7f7f357fa640 10 client.4261.objecter ms_dispatch 0x7f7f38000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:41.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.436+0000 7f7f357fa640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:41.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.436+0000 7f7f357fa640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:41.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.436+0000 7f7f357fa640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-29T11:01:41.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.436+0000 7f7f3cbdf640 10 monclient: get_auth_request con 0x7f7f1c06dc90 auth_method 0 2024-03-29T11:01:41.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.436+0000 7f7f22ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:41.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.436+0000 7f7f22ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:41.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.436+0000 7f7f22ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:01:41.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.436+0000 7f7f22ffd640 1 -- 172.21.15.71:0/1718483667 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f3812ce90 con 0x7f7f380af900 2024-03-29T11:01:41.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.437+0000 7f7f357fa640 10 client.4261.objecter ms_handle_connect 0x7f7f1c06dc90 2024-03-29T11:01:41.426 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.445+0000 7f7f357fa640 1 -- 172.21.15.71:0/1718483667 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7f240509e0 con 0x7f7f380af900 2024-03-29T11:01:41.426 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.445+0000 7f7f357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:41.426 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.445+0000 7f7f357fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:41.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.782+0000 7f7f22ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:41.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.782+0000 7f7f22ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:41.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.782+0000 7f7f22ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:01:41.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.782+0000 7f7f22ffd640 1 -- 172.21.15.71:0/1718483667 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f3812ce90 con 0x7f7f380af900 2024-03-29T11:01:41.766 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:01:41.766 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:41.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.784+0000 7f7f357fa640 1 -- 172.21.15.71:0/1718483667 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f7f3812ce90 con 0x7f7f380af900 2024-03-29T11:01:41.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.784+0000 7f7f357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:41.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.784+0000 7f7f357fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:41.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.787+0000 7f7f22ffd640 10 librados: watch_flush enter 2024-03-29T11:01:41.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.787+0000 7f7f22ffd640 10 librados: watch_flush exit 2024-03-29T11:01:41.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.787+0000 7f7f22ffd640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-29T11:01:41.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.787+0000 7f7f22ffd640 10 client.4261.objecter successfully canceled tick 2024-03-29T11:01:41.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.788+0000 7f7f22ffd640 1 -- 172.21.15.71:0/1718483667 >> v1:172.21.15.137:6816/973332665 conn(0x7f7f1c06dc90 legacy=0x7f7f1c070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:41.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.788+0000 7f7f22ffd640 10 monclient: shutdown 2024-03-29T11:01:41.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.788+0000 7f7f22ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:41.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.788+0000 7f7f22ffd640 1 -- 172.21.15.71:0/1718483667 >> v1:172.21.15.137:6789/0 conn(0x7f7f380af900 legacy=0x7f7f38071610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:41.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.789+0000 7f7f22ffd640 1 -- 172.21.15.71:0/1718483667 shutdown_connections 2024-03-29T11:01:41.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.789+0000 7f7f22ffd640 1 -- 172.21.15.71:0/1718483667 >> 172.21.15.71:0/1718483667 conn(0x7f7f380b35e0 msgr2=0x7f7f38153cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:41.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.789+0000 7f7f22ffd640 1 -- 172.21.15.71:0/1718483667 shutdown_connections 2024-03-29T11:01:41.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.789+0000 7f7f22ffd640 1 -- 172.21.15.71:0/1718483667 wait complete. 2024-03-29T11:01:41.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:41.789+0000 7f7f22ffd640 1 librados: shutdown 2024-03-29T11:01:41.792 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:48.599 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:48.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.783+0000 7f423c4a1640 10 monclient: get_monmap_and_config 2024-03-29T11:01:48.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.783+0000 7f423c4a1640 10 monclient: build_initial_monmap 2024-03-29T11:01:48.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.783+0000 7f423c4a1640 10 monclient: monmap: 2024-03-29T11:01:48.763 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:01:48.763 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:48.763 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:01:48.783968+0000 2024-03-29T11:01:48.763 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:01:48.783968+0000 2024-03-29T11:01:48.763 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:48.763 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:48.763 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:01:48.763 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:01:48.763 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:01:48.763 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:48.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.784+0000 7f423c4a1640 1 Processor -- start 2024-03-29T11:01:48.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.784+0000 7f423c4a1640 1 -- start start 2024-03-29T11:01:48.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.784+0000 7f423c4a1640 10 monclient: init 2024-03-29T11:01:48.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.785+0000 7f423c4a1640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:48.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.785+0000 7f423c4a1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:01:48.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.785+0000 7f423c4a1640 10 monclient(hunting): picked mon.noname-a con 0x7f42341457f0 addr v1:172.21.15.71:6789/0 2024-03-29T11:01:48.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.785+0000 7f423c4a1640 10 monclient(hunting): picked mon.noname-c con 0x7f423413f650 addr v1:172.21.15.137:6789/0 2024-03-29T11:01:48.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.785+0000 7f423aa3e640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f42340a9d70 0x7f423414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:57586/0 (socket says 172.21.15.71:57586) 2024-03-29T11:01:48.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.785+0000 7f423aa3e640 1 -- 172.21.15.71:0/873649499 learned_addr learned my addr 172.21.15.71:0/873649499 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:01:48.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.785+0000 7f423c4a1640 10 monclient(hunting): picked mon.noname-b con 0x7f42340a9d70 addr v1:172.21.15.71:6790/0 2024-03-29T11:01:48.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.785+0000 7f423c4a1640 1 -- 172.21.15.71:0/873649499 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42340aaef0 con 0x7f42341457f0 2024-03-29T11:01:48.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.785+0000 7f423c4a1640 1 -- 172.21.15.71:0/873649499 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4234145bd0 con 0x7f423413f650 2024-03-29T11:01:48.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.785+0000 7f423c4a1640 1 -- 172.21.15.71:0/873649499 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4234145eb0 con 0x7f42340a9d70 2024-03-29T11:01:48.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.785+0000 7f423c4a1640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:48.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.785+0000 7f423c4a1640 10 monclient(hunting): authenticate will time out at 865.043896s 2024-03-29T11:01:48.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.786+0000 7f423923b640 1 -- 172.21.15.71:0/873649499 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073437171 0 0) 0x7f4234145eb0 con 0x7f42340a9d70 2024-03-29T11:01:48.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.786+0000 7f423923b640 10 monclient(hunting): _negotiate 2024-03-29T11:01:48.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.786+0000 7f423923b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:48.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.786+0000 7f423923b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:48.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.786+0000 7f423923b640 10 monclient(hunting): my global_id is 4268 2024-03-29T11:01:48.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.786+0000 7f423923b640 1 -- 172.21.15.71:0/873649499 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4220003520 con 0x7f42340a9d70 2024-03-29T11:01:48.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.786+0000 7f423923b640 1 -- 172.21.15.71:0/873649499 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 35712175 0 0) 0x7f4234145bd0 con 0x7f423413f650 2024-03-29T11:01:48.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.786+0000 7f423923b640 10 monclient(hunting): _negotiate 2024-03-29T11:01:48.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.786+0000 7f423923b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:48.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.786+0000 7f423923b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:48.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.786+0000 7f423923b640 10 monclient(hunting): my global_id is 4264 2024-03-29T11:01:48.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.786+0000 7f423923b640 1 -- 172.21.15.71:0/873649499 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4234145eb0 con 0x7f423413f650 2024-03-29T11:01:48.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.786+0000 7f423923b640 1 -- 172.21.15.71:0/873649499 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1680277434 0 0) 0x7f4220003520 con 0x7f42340a9d70 2024-03-29T11:01:48.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.786+0000 7f423923b640 1 -- 172.21.15.71:0/873649499 >> v1:172.21.15.137:6789/0 conn(0x7f423413f650 legacy=0x7f42340b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:48.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.786+0000 7f423923b640 1 -- 172.21.15.71:0/873649499 >> v1:172.21.15.71:6789/0 conn(0x7f42341457f0 legacy=0x7f423413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:48.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.786+0000 7f423923b640 10 monclient: _finish_hunting 0 2024-03-29T11:01:48.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.786+0000 7f423923b640 1 monclient: found mon.noname-b 2024-03-29T11:01:48.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.786+0000 7f423923b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:48.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.786+0000 7f423923b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:01:48.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.786+0000 7f423923b640 1 -- 172.21.15.71:0/873649499 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4234146340 con 0x7f42340a9d70 2024-03-29T11:01:48.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.786+0000 7f423923b640 10 monclient: _finish_auth 0 2024-03-29T11:01:48.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.786+0000 7f423923b640 10 monclient: _check_auth_tickets 2024-03-29T11:01:48.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.786+0000 7f423923b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:48.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.786+0000 7f423923b640 1 -- 172.21.15.71:0/873649499 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f422c001a10 con 0x7f42340a9d70 2024-03-29T11:01:48.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.786+0000 7f423923b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:48.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.787+0000 7f423923b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:48.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.787+0000 7f423923b640 10 monclient: dump: 2024-03-29T11:01:48.769 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:48.769 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:48.769 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:48.769 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:48.769 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:48.769 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:48.769 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:48.769 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:48.769 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:48.769 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:48.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.787+0000 7f423c4a1640 5 monclient: authenticate success, global_id 4268 2024-03-29T11:01:48.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.787+0000 7f423c4a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:48.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.787+0000 7f423923b640 1 -- 172.21.15.71:0/873649499 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f422c001c30 con 0x7f42340a9d70 2024-03-29T11:01:48.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.787+0000 7f423923b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:48.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.787+0000 7f423c4a1640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:48.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.787+0000 7f423923b640 1 -- 172.21.15.71:0/873649499 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f422c001fc0 con 0x7f42340a9d70 2024-03-29T11:01:48.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.787+0000 7f423923b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:48.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.787+0000 7f423c4a1640 4 set_mon_vals no callback set 2024-03-29T11:01:48.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.787+0000 7f423923b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:48.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.787+0000 7f423923b640 10 monclient: dump: 2024-03-29T11:01:48.770 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:48.770 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:48.770 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:48.770 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:48.770 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:48.770 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:48.770 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:48.770 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:48.771 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:48.771 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:48.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.788+0000 7f423c4a1640 10 monclient: shutdown 2024-03-29T11:01:48.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.788+0000 7f423c4a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:48.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.788+0000 7f423c4a1640 1 -- 172.21.15.71:0/873649499 >> v1:172.21.15.71:6790/0 conn(0x7f42340a9d70 legacy=0x7f423414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:48.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.788+0000 7f423c4a1640 1 -- 172.21.15.71:0/873649499 shutdown_connections 2024-03-29T11:01:48.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.788+0000 7f423c4a1640 1 -- 172.21.15.71:0/873649499 >> 172.21.15.71:0/873649499 conn(0x7f4234066a00 msgr2=0x7f4234066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:48.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.788+0000 7f423c4a1640 1 -- 172.21.15.71:0/873649499 shutdown_connections 2024-03-29T11:01:48.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.788+0000 7f423c4a1640 1 -- 172.21.15.71:0/873649499 wait complete. 2024-03-29T11:01:48.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.789+0000 7f423c4a1640 10 monclient: build_initial_monmap 2024-03-29T11:01:48.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.789+0000 7f423c4a1640 10 monclient: monmap: 2024-03-29T11:01:48.775 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:01:48.775 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:48.775 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:01:48.775 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:01:48.775 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:48.775 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:48.775 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:01:48.775 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:01:48.775 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:01:48.775 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:48.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.789+0000 7f423c4a1640 1 librados: starting msgr at 2024-03-29T11:01:48.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.789+0000 7f423c4a1640 1 librados: starting objecter 2024-03-29T11:01:48.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.789+0000 7f423c4a1640 1 Processor -- start 2024-03-29T11:01:48.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.790+0000 7f423c4a1640 1 -- start start 2024-03-29T11:01:48.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.790+0000 7f423c4a1640 1 librados: setting wanted keys 2024-03-29T11:01:48.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.790+0000 7f423c4a1640 1 librados: calling monclient init 2024-03-29T11:01:48.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.790+0000 7f423c4a1640 10 monclient: init 2024-03-29T11:01:48.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.790+0000 7f423c4a1640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:48.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.790+0000 7f423c4a1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:01:48.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.790+0000 7f423c4a1640 10 monclient(hunting): picked mon.noname-b con 0x7f42340a9d70 addr v1:172.21.15.137:6789/0 2024-03-29T11:01:48.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.790+0000 7f423c4a1640 10 monclient(hunting): picked mon.noname-a con 0x7f423413f650 addr v1:172.21.15.71:6789/0 2024-03-29T11:01:48.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.791+0000 7f423c4a1640 10 monclient(hunting): picked mon.noname-c con 0x7f42341457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:01:48.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.791+0000 7f423c4a1640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42341dba20 con 0x7f423413f650 2024-03-29T11:01:48.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.791+0000 7f423c4a1640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42341dc8a0 con 0x7f42340a9d70 2024-03-29T11:01:48.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.791+0000 7f423c4a1640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f423406f0f0 con 0x7f42341457f0 2024-03-29T11:01:48.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.791+0000 7f423c4a1640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:48.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.791+0000 7f423c4a1640 10 monclient(hunting): authenticate will time out at 865.049140s 2024-03-29T11:01:48.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.791+0000 7f4239a3c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f423413f650 0x7f42341375c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:50866/0 (socket says 172.21.15.71:50866) 2024-03-29T11:01:48.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.791+0000 7f4239a3c640 1 -- 172.21.15.71:0/2908885085 learned_addr learned my addr 172.21.15.71:0/2908885085 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:01:48.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.791+0000 7f422affd640 10 client.?.objecter ms_handle_connect 0x7f423413f650 2024-03-29T11:01:48.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.791+0000 7f422affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:48.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.791+0000 7f422affd640 10 client.?.objecter ms_handle_connect 0x7f42341457f0 2024-03-29T11:01:48.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.791+0000 7f422affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:48.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.791+0000 7f422affd640 1 -- 172.21.15.71:0/2908885085 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1390939432 0 0) 0x7f42341dba20 con 0x7f423413f650 2024-03-29T11:01:48.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.791+0000 7f422affd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:48.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.791+0000 7f422affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:48.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.791+0000 7f422affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:48.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.791+0000 7f422affd640 10 monclient(hunting): my global_id is 4293 2024-03-29T11:01:48.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.791+0000 7f422affd640 1 -- 172.21.15.71:0/2908885085 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4218004eb0 con 0x7f423413f650 2024-03-29T11:01:48.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.791+0000 7f422affd640 10 client.?.objecter ms_handle_connect 0x7f42340a9d70 2024-03-29T11:01:48.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.791+0000 7f422affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:48.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.792+0000 7f422affd640 1 -- 172.21.15.71:0/2908885085 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1574627092 0 0) 0x7f42341dc8a0 con 0x7f42340a9d70 2024-03-29T11:01:48.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.792+0000 7f422affd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:48.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.792+0000 7f422affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:48.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.792+0000 7f422affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:48.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.792+0000 7f422affd640 10 monclient(hunting): my global_id is 4267 2024-03-29T11:01:48.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.792+0000 7f422affd640 1 -- 172.21.15.71:0/2908885085 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42341dba20 con 0x7f42340a9d70 2024-03-29T11:01:48.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.792+0000 7f422affd640 1 -- 172.21.15.71:0/2908885085 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1241011369 0 0) 0x7f4218004eb0 con 0x7f423413f650 2024-03-29T11:01:48.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.792+0000 7f422affd640 1 -- 172.21.15.71:0/2908885085 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42341dc8a0 con 0x7f423413f650 2024-03-29T11:01:48.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.792+0000 7f422affd640 1 -- 172.21.15.71:0/2908885085 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4230001a10 con 0x7f423413f650 2024-03-29T11:01:48.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.792+0000 7f422affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:48.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.792+0000 7f422affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:48.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.792+0000 7f422affd640 10 monclient(hunting): dump: 2024-03-29T11:01:48.778 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:48.778 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:48.778 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:48.778 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:48.779 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:48.779 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:48.779 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:48.779 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:48.779 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:48.779 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:48.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.792+0000 7f422affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:48.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.792+0000 7f422affd640 1 -- 172.21.15.71:0/2908885085 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2913297657 0 0) 0x7f42341dba20 con 0x7f42340a9d70 2024-03-29T11:01:48.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.792+0000 7f422affd640 1 -- 172.21.15.71:0/2908885085 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4218004eb0 con 0x7f42340a9d70 2024-03-29T11:01:48.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.792+0000 7f422affd640 1 -- 172.21.15.71:0/2908885085 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4224001790 con 0x7f42340a9d70 2024-03-29T11:01:48.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.792+0000 7f422affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:48.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.792+0000 7f422affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:48.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.792+0000 7f422affd640 10 monclient(hunting): dump: 2024-03-29T11:01:48.779 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:48.779 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:48.780 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:48.780 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:48.780 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:48.780 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:48.780 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:48.780 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:48.780 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:48.780 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:48.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.792+0000 7f422affd640 1 -- 172.21.15.71:0/2908885085 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2388759862 0 0) 0x7f423406f0f0 con 0x7f42341457f0 2024-03-29T11:01:48.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.792+0000 7f422affd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:48.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.792+0000 7f422affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:48.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.792+0000 7f422affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:48.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.792+0000 7f422affd640 10 monclient(hunting): my global_id is 4271 2024-03-29T11:01:48.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.792+0000 7f422affd640 1 -- 172.21.15.71:0/2908885085 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42341dba20 con 0x7f42341457f0 2024-03-29T11:01:48.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.793+0000 7f422affd640 1 -- 172.21.15.71:0/2908885085 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2717347796 0 0) 0x7f4218004eb0 con 0x7f42340a9d70 2024-03-29T11:01:48.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.793+0000 7f422affd640 1 -- 172.21.15.71:0/2908885085 >> v1:172.21.15.71:6790/0 conn(0x7f42341457f0 legacy=0x7f42341e7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:48.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.793+0000 7f422affd640 1 -- 172.21.15.71:0/2908885085 >> v1:172.21.15.71:6789/0 conn(0x7f423413f650 legacy=0x7f42341375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:48.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.793+0000 7f422affd640 10 monclient: _finish_hunting 0 2024-03-29T11:01:48.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.793+0000 7f422affd640 1 monclient: found mon.b 2024-03-29T11:01:48.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.793+0000 7f422affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:48.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.793+0000 7f422affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:01:48.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.793+0000 7f422affd640 1 -- 172.21.15.71:0/2908885085 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423406f3d0 con 0x7f42340a9d70 2024-03-29T11:01:48.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.793+0000 7f422affd640 10 monclient: _finish_auth 0 2024-03-29T11:01:48.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.793+0000 7f422affd640 10 monclient: _check_auth_tickets 2024-03-29T11:01:48.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.793+0000 7f422affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:48.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.793+0000 7f423c4a1640 5 monclient: authenticate success, global_id 4267 2024-03-29T11:01:48.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.793+0000 7f423c4a1640 10 monclient: _renew_subs 2024-03-29T11:01:48.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.793+0000 7f423c4a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:01:48.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.793+0000 7f423c4a1640 1 -- 172.21.15.71:0/2908885085 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f423406f6b0 con 0x7f42340a9d70 2024-03-29T11:01:48.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.793+0000 7f423c4a1640 10 client.4267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:48.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.793+0000 7f423c4a1640 10 monclient: _renew_subs 2024-03-29T11:01:48.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.793+0000 7f423c4a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:01:48.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.793+0000 7f423c4a1640 1 -- 172.21.15.71:0/2908885085 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f423413bb70 con 0x7f42340a9d70 2024-03-29T11:01:48.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.793+0000 7f423c4a1640 1 librados: init done 2024-03-29T11:01:48.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.793+0000 7f422affd640 1 -- 172.21.15.71:0/2908885085 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42240018f0 con 0x7f42340a9d70 2024-03-29T11:01:48.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.794+0000 7f422affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:48.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.794+0000 7f422affd640 1 -- 172.21.15.71:0/2908885085 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4224003790 con 0x7f42340a9d70 2024-03-29T11:01:48.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.794+0000 7f422affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:48.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.794+0000 7f422affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:48.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.794+0000 7f422affd640 10 monclient: dump: 2024-03-29T11:01:48.782 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:48.782 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:48.782 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:48.782 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:48.782 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:48.783 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:48.783 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:48.783 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:48.783 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:48.783 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:48.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.794+0000 7f422bfff640 4 set_mon_vals no callback set 2024-03-29T11:01:48.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.795+0000 7f423c4a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:48.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.795+0000 7f423c4a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:48.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.795+0000 7f423c4a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:01:48.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.795+0000 7f423c4a1640 1 -- 172.21.15.71:0/2908885085 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f421c005080 con 0x7f42340a9d70 2024-03-29T11:01:48.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.796+0000 7f422affd640 1 -- 172.21.15.71:0/2908885085 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f42240038f0 con 0x7f42340a9d70 2024-03-29T11:01:48.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.797+0000 7f422affd640 1 -- 172.21.15.71:0/2908885085 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f4224083680 con 0x7f42340a9d70 2024-03-29T11:01:48.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.797+0000 7f422affd640 10 client.4267.objecter ms_dispatch 0x7f4234000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:48.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.797+0000 7f422affd640 3 client.4267.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:48.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.797+0000 7f422affd640 3 client.4267.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:48.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.797+0000 7f422affd640 20 client.4267.objecter dump_active .. 0 homeless 2024-03-29T11:01:48.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.797+0000 7f4239a3c640 10 monclient: get_auth_request con 0x7f421806dfc0 auth_method 0 2024-03-29T11:01:48.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.798+0000 7f422affd640 10 client.4267.objecter ms_handle_connect 0x7f421806dfc0 2024-03-29T11:01:48.785 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.802+0000 7f422affd640 1 -- 172.21.15.71:0/2908885085 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f422404ef60 con 0x7f42340a9d70 2024-03-29T11:01:48.785 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.803+0000 7f422affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:48.785 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:48.803+0000 7f422affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:49.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:49.172+0000 7f423c4a1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:49.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:49.172+0000 7f423c4a1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:49.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:49.172+0000 7f423c4a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:01:49.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:49.172+0000 7f423c4a1640 1 -- 172.21.15.71:0/2908885085 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f421c005330 con 0x7f42340a9d70 2024-03-29T11:01:49.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:49.173+0000 7f422affd640 1 -- 172.21.15.71:0/2908885085 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f4224050e90 con 0x7f42340a9d70 2024-03-29T11:01:49.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:49.173+0000 7f422affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:49.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:49.173+0000 7f422affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:49.153 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:01:49.153 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:49.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:49.177+0000 7f423c4a1640 10 librados: watch_flush enter 2024-03-29T11:01:49.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:49.177+0000 7f423c4a1640 10 librados: watch_flush exit 2024-03-29T11:01:49.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:49.177+0000 7f423c4a1640 20 client.4267.objecter shutdown clearing up homeless session... 2024-03-29T11:01:49.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:49.177+0000 7f423c4a1640 10 client.4267.objecter successfully canceled tick 2024-03-29T11:01:49.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:49.177+0000 7f423c4a1640 1 -- 172.21.15.71:0/2908885085 >> v1:172.21.15.137:6816/973332665 conn(0x7f421806dfc0 legacy=0x7f4218070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:49.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:49.177+0000 7f423c4a1640 10 monclient: shutdown 2024-03-29T11:01:49.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:49.177+0000 7f423c4a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:49.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:49.177+0000 7f423c4a1640 1 -- 172.21.15.71:0/2908885085 >> v1:172.21.15.137:6789/0 conn(0x7f42340a9d70 legacy=0x7f42341dc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:49.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:49.177+0000 7f423c4a1640 1 -- 172.21.15.71:0/2908885085 shutdown_connections 2024-03-29T11:01:49.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:49.177+0000 7f423c4a1640 1 -- 172.21.15.71:0/2908885085 >> 172.21.15.71:0/2908885085 conn(0x7f4234066a00 msgr2=0x7f42340b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:49.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:49.177+0000 7f423c4a1640 1 -- 172.21.15.71:0/2908885085 shutdown_connections 2024-03-29T11:01:49.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:49.177+0000 7f423c4a1640 1 -- 172.21.15.71:0/2908885085 wait complete. 2024-03-29T11:01:49.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:49.177+0000 7f423c4a1640 1 librados: shutdown 2024-03-29T11:01:49.171 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:55.978 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:56.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.142+0000 7ff53fb5d640 10 monclient: get_monmap_and_config 2024-03-29T11:01:56.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.142+0000 7ff53fb5d640 10 monclient: build_initial_monmap 2024-03-29T11:01:56.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.142+0000 7ff53fb5d640 10 monclient: monmap: 2024-03-29T11:01:56.122 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:01:56.122 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:56.123 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:01:56.143595+0000 2024-03-29T11:01:56.123 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:01:56.143595+0000 2024-03-29T11:01:56.123 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:56.123 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:56.123 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:01:56.123 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:01:56.123 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:01:56.123 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:56.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.143+0000 7ff53fb5d640 1 Processor -- start 2024-03-29T11:01:56.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.143+0000 7ff53fb5d640 1 -- start start 2024-03-29T11:01:56.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.143+0000 7ff53fb5d640 10 monclient: init 2024-03-29T11:01:56.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.144+0000 7ff53fb5d640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:56.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.144+0000 7ff53fb5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:01:56.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.144+0000 7ff53fb5d640 10 monclient(hunting): picked mon.noname-c con 0x7ff5381457f0 addr v1:172.21.15.137:6789/0 2024-03-29T11:01:56.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.144+0000 7ff53fb5d640 10 monclient(hunting): picked mon.noname-a con 0x7ff53813f650 addr v1:172.21.15.71:6789/0 2024-03-29T11:01:56.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.144+0000 7ff53fb5d640 10 monclient(hunting): picked mon.noname-b con 0x7ff5380a9d70 addr v1:172.21.15.71:6790/0 2024-03-29T11:01:56.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.144+0000 7ff53fb5d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5380aaef0 con 0x7ff53813f650 2024-03-29T11:01:56.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.144+0000 7ff53fb5d640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff538145bd0 con 0x7ff5381457f0 2024-03-29T11:01:56.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.144+0000 7ff53fb5d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff538145eb0 con 0x7ff5380a9d70 2024-03-29T11:01:56.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.144+0000 7ff53fb5d640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:56.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.144+0000 7ff53fb5d640 10 monclient(hunting): authenticate will time out at 872.402968s 2024-03-29T11:01:56.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.144+0000 7ff53e0fa640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7ff5380a9d70 0x7ff53814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:47710/0 (socket says 172.21.15.71:47710) 2024-03-29T11:01:56.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.144+0000 7ff53e0fa640 1 -- 172.21.15.71:0/2309062401 learned_addr learned my addr 172.21.15.71:0/2309062401 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:01:56.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.145+0000 7ff53d0f8640 1 -- 172.21.15.71:0/2309062401 >> v1:172.21.15.71:6789/0 conn(0x7ff53813f650 legacy=0x7ff5380b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-29T11:01:56.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.145+0000 7ff53d0f8640 1 -- 172.21.15.71:0/2309062401 >> v1:172.21.15.71:6789/0 conn(0x7ff53813f650 legacy=0x7ff5380b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-29T11:01:56.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.145+0000 7ff53d0f8640 1 --1- 172.21.15.71:0/2309062401 >> v1:172.21.15.71:6789/0 conn(0x7ff53813f650 0x7ff5380b2620 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-29T11:01:56.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.145+0000 7ff53c8f7640 1 -- 172.21.15.71:0/2309062401 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366987059 0 0) 0x7ff538145eb0 con 0x7ff5380a9d70 2024-03-29T11:01:56.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.145+0000 7ff53c8f7640 10 monclient(hunting): _negotiate 2024-03-29T11:01:56.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.145+0000 7ff53c8f7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:56.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.145+0000 7ff53c8f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:56.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.145+0000 7ff53c8f7640 10 monclient(hunting): my global_id is 4274 2024-03-29T11:01:56.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.145+0000 7ff53c8f7640 1 -- 172.21.15.71:0/2309062401 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff524003520 con 0x7ff5380a9d70 2024-03-29T11:01:56.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.145+0000 7ff53c8f7640 1 -- 172.21.15.71:0/2309062401 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 882941147 0 0) 0x7ff524003520 con 0x7ff5380a9d70 2024-03-29T11:01:56.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.145+0000 7ff53c8f7640 1 -- 172.21.15.71:0/2309062401 >> v1:172.21.15.137:6789/0 conn(0x7ff5381457f0 legacy=0x7ff53813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:01:56.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.145+0000 7ff53c8f7640 1 -- 172.21.15.71:0/2309062401 >> v1:172.21.15.71:6789/0 conn(0x7ff53813f650 legacy=0x7ff5380b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:01:56.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.145+0000 7ff53c8f7640 10 monclient: _finish_hunting 0 2024-03-29T11:01:56.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.145+0000 7ff53c8f7640 1 monclient: found mon.noname-b 2024-03-29T11:01:56.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.145+0000 7ff53c8f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:56.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.145+0000 7ff53c8f7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:01:56.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.146+0000 7ff53c8f7640 1 -- 172.21.15.71:0/2309062401 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff538146340 con 0x7ff5380a9d70 2024-03-29T11:01:56.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.146+0000 7ff53c8f7640 10 monclient: _finish_auth 0 2024-03-29T11:01:56.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.146+0000 7ff53c8f7640 10 monclient: _check_auth_tickets 2024-03-29T11:01:56.130 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.146+0000 7ff53c8f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:56.130 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.146+0000 7ff53c8f7640 1 -- 172.21.15.71:0/2309062401 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7ff528001690 con 0x7ff5380a9d70 2024-03-29T11:01:56.130 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.146+0000 7ff53c8f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:56.130 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.146+0000 7ff53c8f7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:56.130 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.146+0000 7ff53c8f7640 10 monclient: dump: 2024-03-29T11:01:56.130 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:56.130 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:56.130 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:56.130 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:56.130 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:56.130 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:56.130 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:56.131 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:56.131 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:56.131 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:56.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.146+0000 7ff53fb5d640 5 monclient: authenticate success, global_id 4274 2024-03-29T11:01:56.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.146+0000 7ff53fb5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:56.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.146+0000 7ff53c8f7640 1 -- 172.21.15.71:0/2309062401 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff528001920 con 0x7ff5380a9d70 2024-03-29T11:01:56.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.146+0000 7ff53c8f7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:56.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.146+0000 7ff53c8f7640 1 -- 172.21.15.71:0/2309062401 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7ff528001cb0 con 0x7ff5380a9d70 2024-03-29T11:01:56.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.146+0000 7ff53fb5d640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:56.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.146+0000 7ff53fb5d640 4 set_mon_vals no callback set 2024-03-29T11:01:56.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.146+0000 7ff53c8f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:56.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.146+0000 7ff53c8f7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:56.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.146+0000 7ff53c8f7640 10 monclient: dump: 2024-03-29T11:01:56.132 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:56.132 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:56.132 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:56.132 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:56.132 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:56.132 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:56.132 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:56.132 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:56.132 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:56.132 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:56.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.147+0000 7ff53fb5d640 10 monclient: shutdown 2024-03-29T11:01:56.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.147+0000 7ff53fb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:56.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.147+0000 7ff53fb5d640 1 -- 172.21.15.71:0/2309062401 >> v1:172.21.15.71:6790/0 conn(0x7ff5380a9d70 legacy=0x7ff53814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:56.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.147+0000 7ff53fb5d640 1 -- 172.21.15.71:0/2309062401 shutdown_connections 2024-03-29T11:01:56.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.147+0000 7ff53fb5d640 1 -- 172.21.15.71:0/2309062401 >> 172.21.15.71:0/2309062401 conn(0x7ff538066a00 msgr2=0x7ff538066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:56.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.147+0000 7ff53fb5d640 1 -- 172.21.15.71:0/2309062401 shutdown_connections 2024-03-29T11:01:56.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.147+0000 7ff53fb5d640 1 -- 172.21.15.71:0/2309062401 wait complete. 2024-03-29T11:01:56.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.148+0000 7ff53fb5d640 10 monclient: build_initial_monmap 2024-03-29T11:01:56.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.148+0000 7ff53fb5d640 10 monclient: monmap: 2024-03-29T11:01:56.137 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:01:56.137 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:56.137 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:01:56.137 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:01:56.137 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:56.137 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:56.137 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:01:56.137 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:01:56.138 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:01:56.138 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:56.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.149+0000 7ff53fb5d640 1 librados: starting msgr at 2024-03-29T11:01:56.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.149+0000 7ff53fb5d640 1 librados: starting objecter 2024-03-29T11:01:56.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.149+0000 7ff53fb5d640 1 Processor -- start 2024-03-29T11:01:56.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.149+0000 7ff53fb5d640 1 -- start start 2024-03-29T11:01:56.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.149+0000 7ff53fb5d640 1 librados: setting wanted keys 2024-03-29T11:01:56.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.149+0000 7ff53fb5d640 1 librados: calling monclient init 2024-03-29T11:01:56.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.149+0000 7ff53fb5d640 10 monclient: init 2024-03-29T11:01:56.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.150+0000 7ff53fb5d640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:56.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.150+0000 7ff53fb5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:01:56.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.150+0000 7ff53fb5d640 10 monclient(hunting): picked mon.noname-a con 0x7ff5380a9d70 addr v1:172.21.15.71:6789/0 2024-03-29T11:01:56.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.150+0000 7ff53fb5d640 10 monclient(hunting): picked mon.noname-b con 0x7ff53813f650 addr v1:172.21.15.137:6789/0 2024-03-29T11:01:56.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.150+0000 7ff53fb5d640 10 monclient(hunting): picked mon.noname-c con 0x7ff5381457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:01:56.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.150+0000 7ff53d8f9640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7ff5380a9d70 0x7ff5381da870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:51372/0 (socket says 172.21.15.71:51372) 2024-03-29T11:01:56.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.150+0000 7ff53d8f9640 1 -- 172.21.15.71:0/1927511537 learned_addr learned my addr 172.21.15.71:0/1927511537 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:01:56.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.150+0000 7ff52e7fc640 10 client.?.objecter ms_handle_connect 0x7ff5380a9d70 2024-03-29T11:01:56.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.150+0000 7ff52e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:56.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.150+0000 7ff53fb5d640 1 -- 172.21.15.71:0/1927511537 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff53813bb10 con 0x7ff5380a9d70 2024-03-29T11:01:56.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.150+0000 7ff53fb5d640 1 -- 172.21.15.71:0/1927511537 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5381db690 con 0x7ff53813f650 2024-03-29T11:01:56.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.150+0000 7ff53fb5d640 1 -- 172.21.15.71:0/1927511537 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff53813aa10 con 0x7ff5381457f0 2024-03-29T11:01:56.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.150+0000 7ff53fb5d640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:56.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.150+0000 7ff53fb5d640 10 monclient(hunting): authenticate will time out at 872.409044s 2024-03-29T11:01:56.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.151+0000 7ff52e7fc640 1 -- 172.21.15.71:0/1927511537 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3787418120 0 0) 0x7ff53813bb10 con 0x7ff5380a9d70 2024-03-29T11:01:56.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.151+0000 7ff52e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:01:56.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.151+0000 7ff52e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:56.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.151+0000 7ff52e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:56.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.151+0000 7ff52e7fc640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:01:56.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.151+0000 7ff52e7fc640 1 -- 172.21.15.71:0/1927511537 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff51c004eb0 con 0x7ff5380a9d70 2024-03-29T11:01:56.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.151+0000 7ff52e7fc640 1 -- 172.21.15.71:0/1927511537 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1963472207 0 0) 0x7ff51c004eb0 con 0x7ff5380a9d70 2024-03-29T11:01:56.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.151+0000 7ff52e7fc640 1 -- 172.21.15.71:0/1927511537 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff53813bb10 con 0x7ff5380a9d70 2024-03-29T11:01:56.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.151+0000 7ff52e7fc640 10 client.?.objecter ms_handle_connect 0x7ff5381457f0 2024-03-29T11:01:56.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.151+0000 7ff52e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:56.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.151+0000 7ff52e7fc640 1 -- 172.21.15.71:0/1927511537 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7ff530004360 con 0x7ff5380a9d70 2024-03-29T11:01:56.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.151+0000 7ff52e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:56.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.151+0000 7ff52e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:56.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.151+0000 7ff52e7fc640 10 monclient(hunting): dump: 2024-03-29T11:01:56.140 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:56.140 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:56.140 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:56.140 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:56.140 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:56.140 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:56.141 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:56.141 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:56.141 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:56.141 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:56.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.151+0000 7ff52e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:56.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff52e7fc640 1 -- 172.21.15.71:0/1927511537 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085231556 0 0) 0x7ff53813aa10 con 0x7ff5381457f0 2024-03-29T11:01:56.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff52e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:01:56.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff52e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:56.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff52e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:56.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff52e7fc640 10 monclient(hunting): my global_id is 4277 2024-03-29T11:01:56.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff52e7fc640 1 -- 172.21.15.71:0/1927511537 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff51c004eb0 con 0x7ff5381457f0 2024-03-29T11:01:56.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff52e7fc640 10 client.?.objecter ms_handle_connect 0x7ff53813f650 2024-03-29T11:01:56.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff52e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:56.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff52e7fc640 1 -- 172.21.15.71:0/1927511537 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2883000470 0 0) 0x7ff53813bb10 con 0x7ff5380a9d70 2024-03-29T11:01:56.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff52e7fc640 1 -- 172.21.15.71:0/1927511537 >> v1:172.21.15.71:6790/0 conn(0x7ff5381457f0 legacy=0x7ff5381e2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:56.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff52e7fc640 1 -- 172.21.15.71:0/1927511537 >> v1:172.21.15.137:6789/0 conn(0x7ff53813f650 legacy=0x7ff5381daf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:56.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff52e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:01:56.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff52e7fc640 1 monclient: found mon.a 2024-03-29T11:01:56.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff52e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:56.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff52e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:01:56.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff52e7fc640 1 -- 172.21.15.71:0/1927511537 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53813acf0 con 0x7ff5380a9d70 2024-03-29T11:01:56.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff52e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:01:56.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff52e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:01:56.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff52e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:56.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff53fb5d640 5 monclient: authenticate success, global_id 4296 2024-03-29T11:01:56.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff53fb5d640 10 monclient: _renew_subs 2024-03-29T11:01:56.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff53fb5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:01:56.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff53fb5d640 1 -- 172.21.15.71:0/1927511537 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff53813afd0 con 0x7ff5380a9d70 2024-03-29T11:01:56.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff53fb5d640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:56.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff53fb5d640 10 monclient: _renew_subs 2024-03-29T11:01:56.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff53fb5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:01:56.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff53fb5d640 1 -- 172.21.15.71:0/1927511537 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff538139c80 con 0x7ff5380a9d70 2024-03-29T11:01:56.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.152+0000 7ff53fb5d640 1 librados: init done 2024-03-29T11:01:56.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.153+0000 7ff52e7fc640 1 -- 172.21.15.71:0/1927511537 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff530001250 con 0x7ff5380a9d70 2024-03-29T11:01:56.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.153+0000 7ff52e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:56.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.153+0000 7ff52e7fc640 1 -- 172.21.15.71:0/1927511537 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7ff530001620 con 0x7ff5380a9d70 2024-03-29T11:01:56.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.153+0000 7ff52e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:56.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.153+0000 7ff52e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:56.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.153+0000 7ff52e7fc640 10 monclient: dump: 2024-03-29T11:01:56.143 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:01:56.143 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:01:56.143 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:56.144 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:01:56.144 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:56.144 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:01:56.144 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:01:56.144 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:01:56.144 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:01:56.144 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:01:56.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.153+0000 7ff52ffff640 4 set_mon_vals no callback set 2024-03-29T11:01:56.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.154+0000 7ff53fb5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:56.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.154+0000 7ff53fb5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:56.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.154+0000 7ff53fb5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:01:56.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.154+0000 7ff53fb5d640 1 -- 172.21.15.71:0/1927511537 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff520005080 con 0x7ff5380a9d70 2024-03-29T11:01:56.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.155+0000 7ff52e7fc640 1 -- 172.21.15.71:0/1927511537 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7ff530001780 con 0x7ff5380a9d70 2024-03-29T11:01:56.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.156+0000 7ff52e7fc640 1 -- 172.21.15.71:0/1927511537 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7ff530084dd0 con 0x7ff5380a9d70 2024-03-29T11:01:56.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.156+0000 7ff52e7fc640 10 client.4296.objecter ms_dispatch 0x7ff538000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:56.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.156+0000 7ff52e7fc640 3 client.4296.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:56.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.156+0000 7ff52e7fc640 3 client.4296.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:56.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.156+0000 7ff52e7fc640 20 client.4296.objecter dump_active .. 0 homeless 2024-03-29T11:01:56.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.156+0000 7ff53d0f8640 10 monclient: get_auth_request con 0x7ff51c06dd00 auth_method 0 2024-03-29T11:01:56.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.156+0000 7ff52e7fc640 10 client.4296.objecter ms_handle_connect 0x7ff51c06dd00 2024-03-29T11:01:56.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.163+0000 7ff52e7fc640 1 -- 172.21.15.71:0/1927511537 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff530050710 con 0x7ff5380a9d70 2024-03-29T11:01:56.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.163+0000 7ff52e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:56.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.163+0000 7ff52e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:56.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.462+0000 7ff53fb5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:56.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.462+0000 7ff53fb5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:56.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.462+0000 7ff53fb5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:01:56.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.462+0000 7ff53fb5d640 1 -- 172.21.15.71:0/1927511537 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff520005330 con 0x7ff5380a9d70 2024-03-29T11:01:56.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.463+0000 7ff52e7fc640 1 -- 172.21.15.71:0/1927511537 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7ff530050710 con 0x7ff5380a9d70 2024-03-29T11:01:56.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.463+0000 7ff52e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:56.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.463+0000 7ff52e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:56.443 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:01:56.443 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:56.446 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.466+0000 7ff53fb5d640 10 librados: watch_flush enter 2024-03-29T11:01:56.446 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.466+0000 7ff53fb5d640 10 librados: watch_flush exit 2024-03-29T11:01:56.446 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.466+0000 7ff53fb5d640 20 client.4296.objecter shutdown clearing up homeless session... 2024-03-29T11:01:56.446 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.466+0000 7ff53fb5d640 10 client.4296.objecter successfully canceled tick 2024-03-29T11:01:56.446 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.466+0000 7ff53fb5d640 1 -- 172.21.15.71:0/1927511537 >> v1:172.21.15.137:6816/973332665 conn(0x7ff51c06dd00 legacy=0x7ff51c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:56.446 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.466+0000 7ff53fb5d640 10 monclient: shutdown 2024-03-29T11:01:56.446 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.466+0000 7ff53fb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:56.447 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.466+0000 7ff53fb5d640 1 -- 172.21.15.71:0/1927511537 >> v1:172.21.15.71:6789/0 conn(0x7ff5380a9d70 legacy=0x7ff5381da870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:56.447 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.466+0000 7ff53fb5d640 1 -- 172.21.15.71:0/1927511537 shutdown_connections 2024-03-29T11:01:56.447 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.466+0000 7ff53fb5d640 1 -- 172.21.15.71:0/1927511537 >> 172.21.15.71:0/1927511537 conn(0x7ff538066a00 msgr2=0x7ff5380b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:56.447 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.466+0000 7ff53fb5d640 1 -- 172.21.15.71:0/1927511537 shutdown_connections 2024-03-29T11:01:56.447 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.466+0000 7ff53fb5d640 1 -- 172.21.15.71:0/1927511537 wait complete. 2024-03-29T11:01:56.447 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:01:56.466+0000 7ff53fb5d640 1 librados: shutdown 2024-03-29T11:01:56.459 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:03.266 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:03.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.412+0000 7f7bd96ae640 10 monclient: get_monmap_and_config 2024-03-29T11:02:03.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.412+0000 7f7bd96ae640 10 monclient: build_initial_monmap 2024-03-29T11:02:03.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.412+0000 7f7bd96ae640 10 monclient: monmap: 2024-03-29T11:02:03.393 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:02:03.394 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:03.394 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:02:03.413606+0000 2024-03-29T11:02:03.394 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:02:03.413606+0000 2024-03-29T11:02:03.394 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:03.395 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:03.395 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:02:03.395 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:02:03.395 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:02:03.395 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:03.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.413+0000 7f7bd96ae640 1 Processor -- start 2024-03-29T11:02:03.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.413+0000 7f7bd96ae640 1 -- start start 2024-03-29T11:02:03.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.413+0000 7f7bd96ae640 10 monclient: init 2024-03-29T11:02:03.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.414+0000 7f7bd96ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:03.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.414+0000 7f7bd96ae640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:02:03.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.414+0000 7f7bd96ae640 10 monclient(hunting): picked mon.noname-a con 0x7f7bd41457f0 addr v1:172.21.15.71:6789/0 2024-03-29T11:02:03.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.414+0000 7f7bd96ae640 10 monclient(hunting): picked mon.noname-b con 0x7f7bd413f650 addr v1:172.21.15.71:6790/0 2024-03-29T11:02:03.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.414+0000 7f7bd27fc640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f7bd413f650 0x7f7bd40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:49598/0 (socket says 172.21.15.71:49598) 2024-03-29T11:02:03.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.414+0000 7f7bd27fc640 1 -- 172.21.15.71:0/1032600386 learned_addr learned my addr 172.21.15.71:0/1032600386 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:02:03.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.414+0000 7f7bd96ae640 10 monclient(hunting): picked mon.noname-c con 0x7f7bd40a9d70 addr v1:172.21.15.137:6789/0 2024-03-29T11:02:03.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.414+0000 7f7bd96ae640 1 -- 172.21.15.71:0/1032600386 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bd40aaef0 con 0x7f7bd41457f0 2024-03-29T11:02:03.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.414+0000 7f7bd96ae640 1 -- 172.21.15.71:0/1032600386 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bd4145bd0 con 0x7f7bd40a9d70 2024-03-29T11:02:03.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.414+0000 7f7bd96ae640 1 -- 172.21.15.71:0/1032600386 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bd4145eb0 con 0x7f7bd413f650 2024-03-29T11:02:03.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.414+0000 7f7bd96ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:03.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.414+0000 7f7bd96ae640 10 monclient(hunting): authenticate will time out at 879.672733s 2024-03-29T11:02:03.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.415+0000 7f7bd1ffb640 1 -- 172.21.15.71:0/1032600386 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144554081 0 0) 0x7f7bd4145eb0 con 0x7f7bd413f650 2024-03-29T11:02:03.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.415+0000 7f7bd1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:02:03.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.415+0000 7f7bd1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:03.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.415+0000 7f7bd1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:03.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.415+0000 7f7bd1ffb640 10 monclient(hunting): my global_id is 4280 2024-03-29T11:02:03.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.415+0000 7f7bd1ffb640 1 -- 172.21.15.71:0/1032600386 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bbc003520 con 0x7f7bd413f650 2024-03-29T11:02:03.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.415+0000 7f7bd1ffb640 1 -- 172.21.15.71:0/1032600386 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676123102 0 0) 0x7f7bd4145bd0 con 0x7f7bd40a9d70 2024-03-29T11:02:03.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.415+0000 7f7bd1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:02:03.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.415+0000 7f7bd1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:03.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.415+0000 7f7bd1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:03.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.415+0000 7f7bd1ffb640 10 monclient(hunting): my global_id is 4273 2024-03-29T11:02:03.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.415+0000 7f7bd1ffb640 1 -- 172.21.15.71:0/1032600386 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bd4145eb0 con 0x7f7bd40a9d70 2024-03-29T11:02:03.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.415+0000 7f7bd1ffb640 1 -- 172.21.15.71:0/1032600386 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3809392210 0 0) 0x7f7bbc003520 con 0x7f7bd413f650 2024-03-29T11:02:03.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.415+0000 7f7bd1ffb640 1 -- 172.21.15.71:0/1032600386 >> v1:172.21.15.137:6789/0 conn(0x7f7bd40a9d70 legacy=0x7f7bd414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:03.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.415+0000 7f7bd1ffb640 1 -- 172.21.15.71:0/1032600386 >> v1:172.21.15.71:6789/0 conn(0x7f7bd41457f0 legacy=0x7f7bd413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:02:03.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.417+0000 7f7bd1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:02:03.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.417+0000 7f7bd1ffb640 1 monclient: found mon.noname-b 2024-03-29T11:02:03.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.417+0000 7f7bd1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:03.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.417+0000 7f7bd1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:02:03.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.417+0000 7f7bd1ffb640 1 -- 172.21.15.71:0/1032600386 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd4146340 con 0x7f7bd413f650 2024-03-29T11:02:03.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.417+0000 7f7bd1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:02:03.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.417+0000 7f7bd1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:02:03.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.417+0000 7f7bd1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:03.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.417+0000 7f7bd1ffb640 1 -- 172.21.15.71:0/1032600386 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f7bc8001690 con 0x7f7bd413f650 2024-03-29T11:02:03.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.417+0000 7f7bd1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:03.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.417+0000 7f7bd1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:03.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.417+0000 7f7bd1ffb640 10 monclient: dump: 2024-03-29T11:02:03.413 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:03.413 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:03.413 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:03.413 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:03.413 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:03.413 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:03.413 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:03.413 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:03.413 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:03.413 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:03.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.417+0000 7f7bd96ae640 5 monclient: authenticate success, global_id 4280 2024-03-29T11:02:03.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.417+0000 7f7bd96ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:03.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.417+0000 7f7bd1ffb640 1 -- 172.21.15.71:0/1032600386 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7bc80017f0 con 0x7f7bd413f650 2024-03-29T11:02:03.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.417+0000 7f7bd1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:03.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.417+0000 7f7bd1ffb640 1 -- 172.21.15.71:0/1032600386 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f7bc8001c00 con 0x7f7bd413f650 2024-03-29T11:02:03.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.417+0000 7f7bd1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:03.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.417+0000 7f7bd1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:03.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.417+0000 7f7bd1ffb640 10 monclient: dump: 2024-03-29T11:02:03.414 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:03.414 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:03.414 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:03.414 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:03.414 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:03.414 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:03.414 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:03.414 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:03.414 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:03.415 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:03.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.417+0000 7f7bd96ae640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:03.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.417+0000 7f7bd96ae640 4 set_mon_vals no callback set 2024-03-29T11:02:03.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.418+0000 7f7bd96ae640 10 monclient: shutdown 2024-03-29T11:02:03.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.418+0000 7f7bd96ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:03.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.418+0000 7f7bd96ae640 1 -- 172.21.15.71:0/1032600386 >> v1:172.21.15.71:6790/0 conn(0x7f7bd413f650 legacy=0x7f7bd40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:03.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.419+0000 7f7bd96ae640 1 -- 172.21.15.71:0/1032600386 shutdown_connections 2024-03-29T11:02:03.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.419+0000 7f7bd96ae640 1 -- 172.21.15.71:0/1032600386 >> 172.21.15.71:0/1032600386 conn(0x7f7bd4066a00 msgr2=0x7f7bd4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:03.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.419+0000 7f7bd96ae640 1 -- 172.21.15.71:0/1032600386 shutdown_connections 2024-03-29T11:02:03.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.419+0000 7f7bd96ae640 1 -- 172.21.15.71:0/1032600386 wait complete. 2024-03-29T11:02:03.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.420+0000 7f7bd96ae640 10 monclient: build_initial_monmap 2024-03-29T11:02:03.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.420+0000 7f7bd96ae640 10 monclient: monmap: 2024-03-29T11:02:03.416 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:02:03.416 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:03.416 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:02:03.416 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:02:03.416 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:03.416 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:03.416 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:02:03.416 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:02:03.416 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:02:03.417 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:03.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.420+0000 7f7bd96ae640 1 librados: starting msgr at 2024-03-29T11:02:03.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.420+0000 7f7bd96ae640 1 librados: starting objecter 2024-03-29T11:02:03.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.420+0000 7f7bd96ae640 1 Processor -- start 2024-03-29T11:02:03.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.420+0000 7f7bd96ae640 1 -- start start 2024-03-29T11:02:03.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.420+0000 7f7bd96ae640 1 librados: setting wanted keys 2024-03-29T11:02:03.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.420+0000 7f7bd96ae640 1 librados: calling monclient init 2024-03-29T11:02:03.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.420+0000 7f7bd96ae640 10 monclient: init 2024-03-29T11:02:03.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.421+0000 7f7bd96ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:03.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.421+0000 7f7bd96ae640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:02:03.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.421+0000 7f7bd96ae640 10 monclient(hunting): picked mon.noname-a con 0x7f7bd40a9d70 addr v1:172.21.15.71:6789/0 2024-03-29T11:02:03.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.421+0000 7f7bd96ae640 10 monclient(hunting): picked mon.noname-c con 0x7f7bd413f650 addr v1:172.21.15.71:6790/0 2024-03-29T11:02:03.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.421+0000 7f7bd96ae640 10 monclient(hunting): picked mon.noname-b con 0x7f7bd41457f0 addr v1:172.21.15.137:6789/0 2024-03-29T11:02:03.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.421+0000 7f7bd96ae640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bd4137ac0 con 0x7f7bd40a9d70 2024-03-29T11:02:03.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.421+0000 7f7bd96ae640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bd40ac4b0 con 0x7f7bd41457f0 2024-03-29T11:02:03.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.421+0000 7f7bd96ae640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bd40ac790 con 0x7f7bd413f650 2024-03-29T11:02:03.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.421+0000 7f7bd96ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:03.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.421+0000 7f7bd96ae640 10 monclient(hunting): authenticate will time out at 879.679927s 2024-03-29T11:02:03.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.422+0000 7f7bd27fc640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f7bd413f650 0x7f7bd4136ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:49706/0 (socket says 172.21.15.71:49706) 2024-03-29T11:02:03.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.422+0000 7f7bd27fc640 1 -- 172.21.15.71:0/1827583831 learned_addr learned my addr 172.21.15.71:0/1827583831 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:02:03.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.422+0000 7f7bbb7fe640 10 client.?.objecter ms_handle_connect 0x7f7bd413f650 2024-03-29T11:02:03.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.422+0000 7f7bbb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:03.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.422+0000 7f7bbb7fe640 10 client.?.objecter ms_handle_connect 0x7f7bd41457f0 2024-03-29T11:02:03.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.422+0000 7f7bbb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:03.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.423+0000 7f7bbb7fe640 1 -- 172.21.15.71:0/1827583831 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307879168 0 0) 0x7f7bd40ac790 con 0x7f7bd413f650 2024-03-29T11:02:03.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.423+0000 7f7bbb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:02:03.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.423+0000 7f7bbb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:03.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.423+0000 7f7bbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:03.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.423+0000 7f7bbb7fe640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:02:03.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.423+0000 7f7bbb7fe640 1 -- 172.21.15.71:0/1827583831 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bc0004eb0 con 0x7f7bd413f650 2024-03-29T11:02:03.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.423+0000 7f7bbb7fe640 1 -- 172.21.15.71:0/1827583831 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1897488094 0 0) 0x7f7bd40ac4b0 con 0x7f7bd41457f0 2024-03-29T11:02:03.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.423+0000 7f7bbb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:02:03.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.423+0000 7f7bbb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:03.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.423+0000 7f7bbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:03.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.423+0000 7f7bbb7fe640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:02:03.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.423+0000 7f7bbb7fe640 1 -- 172.21.15.71:0/1827583831 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bd40ac790 con 0x7f7bd41457f0 2024-03-29T11:02:03.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.423+0000 7f7bbb7fe640 1 -- 172.21.15.71:0/1827583831 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4007685929 0 0) 0x7f7bc0004eb0 con 0x7f7bd413f650 2024-03-29T11:02:03.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.423+0000 7f7bbb7fe640 1 -- 172.21.15.71:0/1827583831 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7bd40ac4b0 con 0x7f7bd413f650 2024-03-29T11:02:03.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.423+0000 7f7bbb7fe640 1 -- 172.21.15.71:0/1827583831 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f7bc8001bc0 con 0x7f7bd413f650 2024-03-29T11:02:03.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.423+0000 7f7bbb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:03.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.423+0000 7f7bbb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:03.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.423+0000 7f7bbb7fe640 10 monclient(hunting): dump: 2024-03-29T11:02:03.420 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:03.420 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:03.420 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:03.420 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:03.420 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:03.420 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:03.420 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:03.420 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:03.420 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:03.420 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:03.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.423+0000 7f7bbb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:03.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.423+0000 7f7bbb7fe640 1 -- 172.21.15.71:0/1827583831 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4120582107 0 0) 0x7f7bd40ac4b0 con 0x7f7bd413f650 2024-03-29T11:02:03.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.423+0000 7f7bbb7fe640 1 -- 172.21.15.71:0/1827583831 >> v1:172.21.15.137:6789/0 conn(0x7f7bd41457f0 legacy=0x7f7bd41373b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:03.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.423+0000 7f7bbb7fe640 1 -- 172.21.15.71:0/1827583831 >> v1:172.21.15.71:6789/0 conn(0x7f7bd40a9d70 legacy=0x7f7bd4142600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:02:03.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.424+0000 7f7bbb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:02:03.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.424+0000 7f7bbb7fe640 1 monclient: found mon.c 2024-03-29T11:02:03.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.424+0000 7f7bbb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:03.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.424+0000 7f7bbb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:02:03.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.424+0000 7f7bbb7fe640 1 -- 172.21.15.71:0/1827583831 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd4142090 con 0x7f7bd413f650 2024-03-29T11:02:03.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.424+0000 7f7bbb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:02:03.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.424+0000 7f7bbb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:02:03.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.424+0000 7f7bbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:03.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.424+0000 7f7bd96ae640 5 monclient: authenticate success, global_id 4283 2024-03-29T11:02:03.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.424+0000 7f7bd96ae640 10 monclient: _renew_subs 2024-03-29T11:02:03.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.424+0000 7f7bd96ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:02:03.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.424+0000 7f7bd96ae640 1 -- 172.21.15.71:0/1827583831 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bd40ab600 con 0x7f7bd413f650 2024-03-29T11:02:03.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.424+0000 7f7bd96ae640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:03.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.424+0000 7f7bd96ae640 10 monclient: _renew_subs 2024-03-29T11:02:03.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.424+0000 7f7bd96ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:02:03.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.424+0000 7f7bd96ae640 1 -- 172.21.15.71:0/1827583831 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bd413fda0 con 0x7f7bd413f650 2024-03-29T11:02:03.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.424+0000 7f7bd96ae640 1 librados: init done 2024-03-29T11:02:03.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.425+0000 7f7bbb7fe640 1 -- 172.21.15.71:0/1827583831 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7bc8004a90 con 0x7f7bd413f650 2024-03-29T11:02:03.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.425+0000 7f7bbb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:03.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.425+0000 7f7bbb7fe640 1 -- 172.21.15.71:0/1827583831 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f7bc8004ea0 con 0x7f7bd413f650 2024-03-29T11:02:03.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.425+0000 7f7bbb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:03.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.425+0000 7f7bbb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:03.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.425+0000 7f7bbb7fe640 10 monclient: dump: 2024-03-29T11:02:03.422 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:03.422 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:03.422 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:03.422 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:03.422 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:03.423 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:03.423 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:03.423 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:03.423 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:03.423 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:03.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.425+0000 7f7bd17fa640 4 set_mon_vals no callback set 2024-03-29T11:02:03.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.426+0000 7f7bd96ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:03.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.426+0000 7f7bd96ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:03.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.426+0000 7f7bd96ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:02:03.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.426+0000 7f7bd96ae640 1 -- 172.21.15.71:0/1827583831 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb4005080 con 0x7f7bd413f650 2024-03-29T11:02:03.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.427+0000 7f7bbb7fe640 1 -- 172.21.15.71:0/1827583831 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f7bc8005000 con 0x7f7bd413f650 2024-03-29T11:02:03.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.428+0000 7f7bbb7fe640 1 -- 172.21.15.71:0/1827583831 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f7bc8084c30 con 0x7f7bd413f650 2024-03-29T11:02:03.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.428+0000 7f7bbb7fe640 10 client.4283.objecter ms_dispatch 0x7f7bd4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:02:03.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.428+0000 7f7bbb7fe640 3 client.4283.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:02:03.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.428+0000 7f7bbb7fe640 3 client.4283.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:02:03.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.428+0000 7f7bbb7fe640 20 client.4283.objecter dump_active .. 0 homeless 2024-03-29T11:02:03.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.428+0000 7f7bd2ffd640 10 monclient: get_auth_request con 0x7f7bc006dd00 auth_method 0 2024-03-29T11:02:03.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.428+0000 7f7bbb7fe640 10 client.4283.objecter ms_handle_connect 0x7f7bc006dd00 2024-03-29T11:02:03.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.435+0000 7f7bbb7fe640 1 -- 172.21.15.71:0/1827583831 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7bc8050510 con 0x7f7bd413f650 2024-03-29T11:02:03.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.435+0000 7f7bbb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:03.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.435+0000 7f7bbb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:03.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.719+0000 7f7bd96ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:03.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.719+0000 7f7bd96ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:03.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.719+0000 7f7bd96ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:02:03.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.719+0000 7f7bd96ae640 1 -- 172.21.15.71:0/1827583831 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7bb4005330 con 0x7f7bd413f650 2024-03-29T11:02:03.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.719+0000 7f7bbb7fe640 1 -- 172.21.15.71:0/1827583831 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f7bc8052440 con 0x7f7bd413f650 2024-03-29T11:02:03.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.719+0000 7f7bbb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:03.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.719+0000 7f7bbb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:03.701 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:02:03.702 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:03.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.722+0000 7f7bd96ae640 10 librados: watch_flush enter 2024-03-29T11:02:03.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.722+0000 7f7bd96ae640 10 librados: watch_flush exit 2024-03-29T11:02:03.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.722+0000 7f7bd96ae640 20 client.4283.objecter shutdown clearing up homeless session... 2024-03-29T11:02:03.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.722+0000 7f7bd96ae640 10 client.4283.objecter successfully canceled tick 2024-03-29T11:02:03.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.722+0000 7f7bd96ae640 1 -- 172.21.15.71:0/1827583831 >> v1:172.21.15.137:6816/973332665 conn(0x7f7bc006dd00 legacy=0x7f7bc00701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:03.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.722+0000 7f7bd96ae640 10 monclient: shutdown 2024-03-29T11:02:03.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.722+0000 7f7bd96ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:03.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.722+0000 7f7bd96ae640 1 -- 172.21.15.71:0/1827583831 >> v1:172.21.15.71:6790/0 conn(0x7f7bd413f650 legacy=0x7f7bd4136ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:03.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.722+0000 7f7bd96ae640 1 -- 172.21.15.71:0/1827583831 shutdown_connections 2024-03-29T11:02:03.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.722+0000 7f7bd96ae640 1 -- 172.21.15.71:0/1827583831 >> 172.21.15.71:0/1827583831 conn(0x7f7bd4066a00 msgr2=0x7f7bd413aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:03.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.722+0000 7f7bd96ae640 1 -- 172.21.15.71:0/1827583831 shutdown_connections 2024-03-29T11:02:03.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.723+0000 7f7bd96ae640 1 -- 172.21.15.71:0/1827583831 wait complete. 2024-03-29T11:02:03.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:03.723+0000 7f7bd96ae640 1 librados: shutdown 2024-03-29T11:02:03.717 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:10.524 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:10.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.696+0000 7f4424cff640 10 monclient: get_monmap_and_config 2024-03-29T11:02:10.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.696+0000 7f4424cff640 10 monclient: build_initial_monmap 2024-03-29T11:02:10.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.696+0000 7f4424cff640 10 monclient: monmap: 2024-03-29T11:02:10.676 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:02:10.677 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:10.677 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:02:10.697688+0000 2024-03-29T11:02:10.677 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:02:10.697688+0000 2024-03-29T11:02:10.677 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:10.677 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:10.677 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:02:10.677 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:02:10.677 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:02:10.677 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:10.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.698+0000 7f4424cff640 1 Processor -- start 2024-03-29T11:02:10.678 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.699+0000 7f4424cff640 1 -- start start 2024-03-29T11:02:10.678 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.699+0000 7f4424cff640 10 monclient: init 2024-03-29T11:02:10.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.699+0000 7f4424cff640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:10.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.699+0000 7f4424cff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:02:10.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.699+0000 7f4424cff640 10 monclient(hunting): picked mon.noname-a con 0x7f44200b0770 addr v1:172.21.15.71:6789/0 2024-03-29T11:02:10.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.699+0000 7f4424cff640 10 monclient(hunting): picked mon.noname-c con 0x7f442014a700 addr v1:172.21.15.137:6789/0 2024-03-29T11:02:10.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.699+0000 7f4424cff640 10 monclient(hunting): picked mon.noname-b con 0x7f44200ac5a0 addr v1:172.21.15.71:6790/0 2024-03-29T11:02:10.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.699+0000 7f4424cff640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44200aba70 con 0x7f44200b0770 2024-03-29T11:02:10.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.700+0000 7f4424cff640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44200aecb0 con 0x7f442014a700 2024-03-29T11:02:10.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.700+0000 7f4424cff640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44200af150 con 0x7f44200ac5a0 2024-03-29T11:02:10.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.700+0000 7f4424cff640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:10.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.700+0000 7f4424cff640 10 monclient(hunting): authenticate will time out at 886.958146s 2024-03-29T11:02:10.682 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.702+0000 7f441e59c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f44200b0770 0x7f4420149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:58717/0 (socket says 172.21.15.71:58717) 2024-03-29T11:02:10.682 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.702+0000 7f441e59c640 1 -- 172.21.15.71:0/3633937647 learned_addr learned my addr 172.21.15.71:0/3633937647 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:02:10.682 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.701+0000 7f441ed9d640 1 --1- 172.21.15.71:0/3633937647 >> v1:172.21.15.71:6790/0 conn(0x7f44200ac5a0 0x7f4420151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:36948/0 (socket says 172.21.15.71:36948) 2024-03-29T11:02:10.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.704+0000 7f441dd9b640 1 -- 172.21.15.71:0/3633937647 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3462577469 0 0) 0x7f44200aba70 con 0x7f44200b0770 2024-03-29T11:02:10.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.704+0000 7f441dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:02:10.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.704+0000 7f441dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:10.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.704+0000 7f441dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:10.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.704+0000 7f441dd9b640 10 monclient(hunting): my global_id is 4299 2024-03-29T11:02:10.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.704+0000 7f441dd9b640 1 -- 172.21.15.71:0/3633937647 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f440c003520 con 0x7f44200b0770 2024-03-29T11:02:10.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.705+0000 7f441dd9b640 1 -- 172.21.15.71:0/3633937647 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1525848946 0 0) 0x7f440c003520 con 0x7f44200b0770 2024-03-29T11:02:10.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.705+0000 7f441dd9b640 1 -- 172.21.15.71:0/3633937647 >> v1:172.21.15.71:6790/0 conn(0x7f44200ac5a0 legacy=0x7f4420151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:10.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.705+0000 7f441dd9b640 1 -- 172.21.15.71:0/3633937647 >> v1:172.21.15.137:6789/0 conn(0x7f442014a700 legacy=0x7f44200abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:02:10.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.706+0000 7f441dd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:02:10.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.706+0000 7f441dd9b640 1 monclient: found mon.noname-a 2024-03-29T11:02:10.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.706+0000 7f441dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:10.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.706+0000 7f441dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:02:10.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.706+0000 7f441dd9b640 1 -- 172.21.15.71:0/3633937647 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44200ac9d0 con 0x7f44200b0770 2024-03-29T11:02:10.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.706+0000 7f441dd9b640 10 monclient: _finish_auth 0 2024-03-29T11:02:10.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.706+0000 7f441dd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:02:10.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.706+0000 7f441dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:10.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.706+0000 7f441dd9b640 1 -- 172.21.15.71:0/3633937647 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4410001690 con 0x7f44200b0770 2024-03-29T11:02:10.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.706+0000 7f441dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:10.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.706+0000 7f441dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:02:10.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.706+0000 7f441dd9b640 10 monclient: dump: 2024-03-29T11:02:10.687 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:10.687 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:10.687 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:10.687 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:10.687 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:10.687 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:10.687 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:10.687 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:10.688 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:10.688 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:10.688 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.707+0000 7f441dd9b640 1 -- 172.21.15.71:0/3633937647 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4410001920 con 0x7f44200b0770 2024-03-29T11:02:10.688 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.707+0000 7f441dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:10.688 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.707+0000 7f441dd9b640 1 -- 172.21.15.71:0/3633937647 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4410001d30 con 0x7f44200b0770 2024-03-29T11:02:10.688 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.707+0000 7f441dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:10.688 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.707+0000 7f441dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:10.688 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.707+0000 7f441dd9b640 10 monclient: dump: 2024-03-29T11:02:10.688 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:10.688 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:10.689 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:10.689 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:10.689 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:10.689 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:10.689 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:10.689 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:10.689 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:10.689 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:10.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.708+0000 7f4424cff640 5 monclient: authenticate success, global_id 4299 2024-03-29T11:02:10.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.708+0000 7f4424cff640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:10.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.708+0000 7f4424cff640 4 set_mon_vals no callback set 2024-03-29T11:02:10.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.709+0000 7f4424cff640 10 monclient: shutdown 2024-03-29T11:02:10.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.709+0000 7f4424cff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:10.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.709+0000 7f4424cff640 1 -- 172.21.15.71:0/3633937647 >> v1:172.21.15.71:6789/0 conn(0x7f44200b0770 legacy=0x7f4420149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:10.691 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.709+0000 7f4424cff640 1 -- 172.21.15.71:0/3633937647 shutdown_connections 2024-03-29T11:02:10.691 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.709+0000 7f4424cff640 1 -- 172.21.15.71:0/3633937647 >> 172.21.15.71:0/3633937647 conn(0x7f442006b660 msgr2=0x7f442006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:10.691 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.710+0000 7f4424cff640 1 -- 172.21.15.71:0/3633937647 shutdown_connections 2024-03-29T11:02:10.692 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.711+0000 7f4424cff640 1 -- 172.21.15.71:0/3633937647 wait complete. 2024-03-29T11:02:10.692 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.712+0000 7f4424cff640 10 monclient: build_initial_monmap 2024-03-29T11:02:10.692 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.712+0000 7f4424cff640 10 monclient: monmap: 2024-03-29T11:02:10.693 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:02:10.693 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:10.693 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:02:10.693 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:02:10.693 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:10.693 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:10.693 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:02:10.693 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:02:10.693 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:02:10.693 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:10.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.712+0000 7f4424cff640 1 librados: starting msgr at 2024-03-29T11:02:10.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.712+0000 7f4424cff640 1 librados: starting objecter 2024-03-29T11:02:10.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.713+0000 7f4424cff640 1 Processor -- start 2024-03-29T11:02:10.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.713+0000 7f4424cff640 1 -- start start 2024-03-29T11:02:10.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.713+0000 7f4424cff640 1 librados: setting wanted keys 2024-03-29T11:02:10.694 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.713+0000 7f4424cff640 1 librados: calling monclient init 2024-03-29T11:02:10.694 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.713+0000 7f4424cff640 10 monclient: init 2024-03-29T11:02:10.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.713+0000 7f4424cff640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:10.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.713+0000 7f4424cff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:02:10.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.713+0000 7f4424cff640 10 monclient(hunting): picked mon.noname-a con 0x7f44200ac5a0 addr v1:172.21.15.71:6789/0 2024-03-29T11:02:10.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.713+0000 7f4424cff640 10 monclient(hunting): picked mon.noname-b con 0x7f44200b0770 addr v1:172.21.15.137:6789/0 2024-03-29T11:02:10.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.713+0000 7f4424cff640 10 monclient(hunting): picked mon.noname-c con 0x7f442014a700 addr v1:172.21.15.71:6790/0 2024-03-29T11:02:10.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.713+0000 7f4424cff640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4420148930 con 0x7f44200ac5a0 2024-03-29T11:02:10.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.713+0000 7f4424cff640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f442011e1b0 con 0x7f44200b0770 2024-03-29T11:02:10.696 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.713+0000 7f4424cff640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f442011cf70 con 0x7f442014a700 2024-03-29T11:02:10.696 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.713+0000 7f4424cff640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:10.696 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.713+0000 7f4424cff640 10 monclient(hunting): authenticate will time out at 886.972087s 2024-03-29T11:02:10.696 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.715+0000 7f441ed9d640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f442014a700 0x7f4420148220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:37760/0 (socket says 172.21.15.71:37760) 2024-03-29T11:02:10.696 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.715+0000 7f441ed9d640 1 -- 172.21.15.71:0/276510055 learned_addr learned my addr 172.21.15.71:0/276510055 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:02:10.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.717+0000 7f4416ffd640 10 client.?.objecter ms_handle_connect 0x7f442014a700 2024-03-29T11:02:10.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.717+0000 7f4416ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:10.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.717+0000 7f4416ffd640 1 -- 172.21.15.71:0/276510055 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153906031 0 0) 0x7f442011cf70 con 0x7f442014a700 2024-03-29T11:02:10.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.717+0000 7f4416ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:10.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.717+0000 7f4416ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:10.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.717+0000 7f4416ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:10.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.717+0000 7f4416ffd640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:02:10.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.717+0000 7f4416ffd640 1 -- 172.21.15.71:0/276510055 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4400004eb0 con 0x7f442014a700 2024-03-29T11:02:10.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.717+0000 7f4416ffd640 1 -- 172.21.15.71:0/276510055 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2477465016 0 0) 0x7f4400004eb0 con 0x7f442014a700 2024-03-29T11:02:10.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.717+0000 7f4416ffd640 1 -- 172.21.15.71:0/276510055 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f442011cf70 con 0x7f442014a700 2024-03-29T11:02:10.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.717+0000 7f4416ffd640 1 -- 172.21.15.71:0/276510055 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4418001cd0 con 0x7f442014a700 2024-03-29T11:02:10.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.717+0000 7f4416ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:10.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.717+0000 7f4416ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:10.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.717+0000 7f4416ffd640 10 monclient(hunting): dump: 2024-03-29T11:02:10.699 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:10.699 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:10.699 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:10.699 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:10.699 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:10.699 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:10.699 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:10.699 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:10.699 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:10.699 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:10.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.717+0000 7f4416ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:10.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.718+0000 7f4416ffd640 10 client.?.objecter ms_handle_connect 0x7f44200ac5a0 2024-03-29T11:02:10.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.718+0000 7f4416ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:10.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.718+0000 7f4416ffd640 1 -- 172.21.15.71:0/276510055 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2235109529 0 0) 0x7f442011cf70 con 0x7f442014a700 2024-03-29T11:02:10.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.718+0000 7f4416ffd640 1 -- 172.21.15.71:0/276510055 >> v1:172.21.15.137:6789/0 conn(0x7f44200b0770 legacy=0x7f4420147b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:02:10.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.719+0000 7f4416ffd640 1 -- 172.21.15.71:0/276510055 >> v1:172.21.15.71:6789/0 conn(0x7f44200ac5a0 legacy=0x7f44201e3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:10.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.719+0000 7f4416ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:02:10.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.719+0000 7f4416ffd640 1 monclient: found mon.c 2024-03-29T11:02:10.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.719+0000 7f4416ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:10.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.719+0000 7f4416ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:02:10.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.719+0000 7f4416ffd640 1 -- 172.21.15.71:0/276510055 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442011e3d0 con 0x7f442014a700 2024-03-29T11:02:10.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.720+0000 7f4416ffd640 10 monclient: _finish_auth 0 2024-03-29T11:02:10.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.720+0000 7f4416ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:02:10.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.720+0000 7f4416ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:10.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.720+0000 7f4424cff640 5 monclient: authenticate success, global_id 4289 2024-03-29T11:02:10.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.720+0000 7f4424cff640 10 monclient: _renew_subs 2024-03-29T11:02:10.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.720+0000 7f4424cff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:02:10.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.720+0000 7f4424cff640 1 -- 172.21.15.71:0/276510055 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f442011e620 con 0x7f442014a700 2024-03-29T11:02:10.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.720+0000 7f4424cff640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:10.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.720+0000 7f4424cff640 10 monclient: _renew_subs 2024-03-29T11:02:10.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.720+0000 7f4424cff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:02:10.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.720+0000 7f4424cff640 1 -- 172.21.15.71:0/276510055 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f442011d950 con 0x7f442014a700 2024-03-29T11:02:10.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.720+0000 7f4424cff640 1 librados: init done 2024-03-29T11:02:10.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.721+0000 7f4416ffd640 1 -- 172.21.15.71:0/276510055 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4418001ff0 con 0x7f442014a700 2024-03-29T11:02:10.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.722+0000 7f4416ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:10.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.722+0000 7f4416ffd640 1 -- 172.21.15.71:0/276510055 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4418004fb0 con 0x7f442014a700 2024-03-29T11:02:10.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.722+0000 7f4416ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:10.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.722+0000 7f441d59a640 4 set_mon_vals no callback set 2024-03-29T11:02:10.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.722+0000 7f4416ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:10.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.722+0000 7f4416ffd640 10 monclient: dump: 2024-03-29T11:02:10.705 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:10.705 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:10.705 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:10.705 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:10.705 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:10.705 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:10.705 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:10.706 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:10.706 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:10.706 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:10.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.722+0000 7f4424cff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:10.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.723+0000 7f4424cff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:10.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.723+0000 7f4424cff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:02:10.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.723+0000 7f4424cff640 1 -- 172.21.15.71:0/276510055 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4404005080 con 0x7f442014a700 2024-03-29T11:02:10.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.725+0000 7f4416ffd640 1 -- 172.21.15.71:0/276510055 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f4418005110 con 0x7f442014a700 2024-03-29T11:02:10.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.726+0000 7f4416ffd640 1 -- 172.21.15.71:0/276510055 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f4418084f10 con 0x7f442014a700 2024-03-29T11:02:10.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.726+0000 7f4416ffd640 10 client.4289.objecter ms_dispatch 0x7f4420000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:02:10.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.726+0000 7f4416ffd640 3 client.4289.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:02:10.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.726+0000 7f4416ffd640 3 client.4289.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:02:10.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.726+0000 7f4416ffd640 20 client.4289.objecter dump_active .. 0 homeless 2024-03-29T11:02:10.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.726+0000 7f441e59c640 10 monclient: get_auth_request con 0x7f440006dac0 auth_method 0 2024-03-29T11:02:10.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.727+0000 7f4416ffd640 10 client.4289.objecter ms_handle_connect 0x7f440006dac0 2024-03-29T11:02:10.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.735+0000 7f4416ffd640 1 -- 172.21.15.71:0/276510055 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f44180507f0 con 0x7f442014a700 2024-03-29T11:02:10.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.735+0000 7f4416ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:10.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:10.735+0000 7f4416ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:11.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:11.072+0000 7f4424cff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:11.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:11.072+0000 7f4424cff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:11.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:11.072+0000 7f4424cff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:02:11.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:11.072+0000 7f4424cff640 1 -- 172.21.15.71:0/276510055 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4404005330 con 0x7f442014a700 2024-03-29T11:02:11.056 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:02:11.056 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:11.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:11.074+0000 7f4416ffd640 1 -- 172.21.15.71:0/276510055 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f4418052720 con 0x7f442014a700 2024-03-29T11:02:11.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:11.074+0000 7f4416ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:11.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:11.074+0000 7f4416ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:11.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:11.078+0000 7f4424cff640 10 librados: watch_flush enter 2024-03-29T11:02:11.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:11.078+0000 7f4424cff640 10 librados: watch_flush exit 2024-03-29T11:02:11.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:11.078+0000 7f4424cff640 20 client.4289.objecter shutdown clearing up homeless session... 2024-03-29T11:02:11.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:11.078+0000 7f4424cff640 10 client.4289.objecter successfully canceled tick 2024-03-29T11:02:11.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:11.078+0000 7f4424cff640 1 -- 172.21.15.71:0/276510055 >> v1:172.21.15.137:6816/973332665 conn(0x7f440006dac0 legacy=0x7f440006ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:11.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:11.078+0000 7f4424cff640 10 monclient: shutdown 2024-03-29T11:02:11.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:11.078+0000 7f4424cff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:11.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:11.078+0000 7f4424cff640 1 -- 172.21.15.71:0/276510055 >> v1:172.21.15.71:6790/0 conn(0x7f442014a700 legacy=0x7f4420148220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:11.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:11.078+0000 7f4424cff640 1 -- 172.21.15.71:0/276510055 shutdown_connections 2024-03-29T11:02:11.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:11.078+0000 7f4424cff640 1 -- 172.21.15.71:0/276510055 >> 172.21.15.71:0/276510055 conn(0x7f442006b660 msgr2=0x7f442006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:11.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:11.079+0000 7f4424cff640 1 -- 172.21.15.71:0/276510055 shutdown_connections 2024-03-29T11:02:11.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:11.079+0000 7f4424cff640 1 -- 172.21.15.71:0/276510055 wait complete. 2024-03-29T11:02:11.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:11.079+0000 7f4424cff640 1 librados: shutdown 2024-03-29T11:02:11.075 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:17.883 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:18.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.048+0000 7f1004b20640 10 monclient: get_monmap_and_config 2024-03-29T11:02:18.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.048+0000 7f1004b20640 10 monclient: build_initial_monmap 2024-03-29T11:02:18.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.048+0000 7f1004b20640 10 monclient: monmap: 2024-03-29T11:02:18.028 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:02:18.028 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:18.028 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:02:18.048922+0000 2024-03-29T11:02:18.028 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:02:18.048922+0000 2024-03-29T11:02:18.028 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:18.028 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:18.028 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:02:18.028 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:02:18.028 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:02:18.028 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:18.029 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.049+0000 7f1004b20640 1 Processor -- start 2024-03-29T11:02:18.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.049+0000 7f1004b20640 1 -- start start 2024-03-29T11:02:18.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.049+0000 7f1004b20640 10 monclient: init 2024-03-29T11:02:18.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.050+0000 7f1004b20640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:18.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.050+0000 7f1004b20640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:02:18.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.050+0000 7f1004b20640 10 monclient(hunting): picked mon.noname-b con 0x7f0ff8006e30 addr v1:172.21.15.71:6790/0 2024-03-29T11:02:18.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.050+0000 7f1004b20640 10 monclient(hunting): picked mon.noname-c con 0x7f0ff8035e00 addr v1:172.21.15.137:6789/0 2024-03-29T11:02:18.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.050+0000 7f1004b20640 10 monclient(hunting): picked mon.noname-a con 0x7f0ff80c9670 addr v1:172.21.15.71:6789/0 2024-03-29T11:02:18.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.050+0000 7f1004b20640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff8035970 con 0x7f0ff80c9670 2024-03-29T11:02:18.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.050+0000 7f1004b20640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff8036280 con 0x7f0ff8035e00 2024-03-29T11:02:18.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.051+0000 7f1004b20640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff8035640 con 0x7f0ff8006e30 2024-03-29T11:02:18.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.051+0000 7f1004b20640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:18.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.051+0000 7f1004b20640 10 monclient(hunting): authenticate will time out at 894.309539s 2024-03-29T11:02:18.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.051+0000 7f0fff7fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f0ff8006e30 0x7f0ff80cbcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:42864/0 (socket says 172.21.15.71:42864) 2024-03-29T11:02:18.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.051+0000 7f0fff7fe640 1 -- 172.21.15.71:0/3385720271 learned_addr learned my addr 172.21.15.71:0/3385720271 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:02:18.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.051+0000 7f0ffe7fc640 1 -- 172.21.15.71:0/3385720271 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3391613236 0 0) 0x7f0ff8035640 con 0x7f0ff8006e30 2024-03-29T11:02:18.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.051+0000 7f0ffe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:02:18.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.052+0000 7f0ffe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:18.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.052+0000 7f0ffe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:18.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.052+0000 7f0ffe7fc640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:02:18.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.052+0000 7f0ffe7fc640 1 -- 172.21.15.71:0/3385720271 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ff0003520 con 0x7f0ff8006e30 2024-03-29T11:02:18.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.052+0000 7f0ffe7fc640 1 -- 172.21.15.71:0/3385720271 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3919007069 0 0) 0x7f0ff0003520 con 0x7f0ff8006e30 2024-03-29T11:02:18.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.052+0000 7f0ffe7fc640 1 -- 172.21.15.71:0/3385720271 >> v1:172.21.15.137:6789/0 conn(0x7f0ff8035e00 legacy=0x7f0ff80c8f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:02:18.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.054+0000 7f0ffe7fc640 1 -- 172.21.15.71:0/3385720271 >> v1:172.21.15.71:6789/0 conn(0x7f0ff80c9670 legacy=0x7f0ff80d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:02:18.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.054+0000 7f0ffe7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:02:18.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.054+0000 7f0ffe7fc640 1 monclient: found mon.noname-b 2024-03-29T11:02:18.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.054+0000 7f0ffe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:18.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.055+0000 7f0ffe7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:02:18.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.055+0000 7f0ffe7fc640 1 -- 172.21.15.71:0/3385720271 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff8034e20 con 0x7f0ff8006e30 2024-03-29T11:02:18.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.055+0000 7f0ffe7fc640 10 monclient: _finish_auth 0 2024-03-29T11:02:18.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.055+0000 7f0ffe7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:02:18.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.055+0000 7f0ffe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:18.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.055+0000 7f1004b20640 5 monclient: authenticate success, global_id 4292 2024-03-29T11:02:18.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.055+0000 7f1004b20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:18.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.055+0000 7f0ffe7fc640 1 -- 172.21.15.71:0/3385720271 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f0fec001690 con 0x7f0ff8006e30 2024-03-29T11:02:18.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.055+0000 7f0ffe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:18.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.055+0000 7f0ffe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:18.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.055+0000 7f0ffe7fc640 10 monclient: dump: 2024-03-29T11:02:18.038 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:18.038 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:18.038 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:18.038 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:18.038 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:18.038 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:18.038 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:18.038 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:18.038 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:18.038 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:18.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.055+0000 7f1004b20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:18.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.055+0000 7f0ffe7fc640 1 -- 172.21.15.71:0/3385720271 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0fec001920 con 0x7f0ff8006e30 2024-03-29T11:02:18.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.056+0000 7f0ffe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:18.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.056+0000 7f1004b20640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:18.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.056+0000 7f1004b20640 4 set_mon_vals no callback set 2024-03-29T11:02:18.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.056+0000 7f0ffe7fc640 1 -- 172.21.15.71:0/3385720271 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f0fec001cf0 con 0x7f0ff8006e30 2024-03-29T11:02:18.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.056+0000 7f0ffe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:18.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.056+0000 7f0ffe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:18.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.056+0000 7f0ffe7fc640 10 monclient: dump: 2024-03-29T11:02:18.039 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:18.039 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:18.039 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:18.039 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:18.039 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:18.039 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:18.039 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:18.039 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:18.039 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:18.040 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:18.040 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.057+0000 7f1004b20640 10 monclient: shutdown 2024-03-29T11:02:18.040 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.057+0000 7f1004b20640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:18.040 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.057+0000 7f1004b20640 1 -- 172.21.15.71:0/3385720271 >> v1:172.21.15.71:6790/0 conn(0x7f0ff8006e30 legacy=0x7f0ff80cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:18.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.057+0000 7f1004b20640 1 -- 172.21.15.71:0/3385720271 shutdown_connections 2024-03-29T11:02:18.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.057+0000 7f1004b20640 1 -- 172.21.15.71:0/3385720271 >> 172.21.15.71:0/3385720271 conn(0x7f0ff8005c00 msgr2=0x7f0ff8006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:18.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.057+0000 7f1004b20640 1 -- 172.21.15.71:0/3385720271 shutdown_connections 2024-03-29T11:02:18.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.057+0000 7f1004b20640 1 -- 172.21.15.71:0/3385720271 wait complete. 2024-03-29T11:02:18.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.058+0000 7f1004b20640 10 monclient: build_initial_monmap 2024-03-29T11:02:18.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.058+0000 7f1004b20640 10 monclient: monmap: 2024-03-29T11:02:18.042 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:02:18.042 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:18.042 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:02:18.043 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:02:18.043 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:18.043 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:18.043 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:02:18.043 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:02:18.043 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:02:18.043 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:18.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.058+0000 7f1004b20640 1 librados: starting msgr at 2024-03-29T11:02:18.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.058+0000 7f1004b20640 1 librados: starting objecter 2024-03-29T11:02:18.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.058+0000 7f1004b20640 1 Processor -- start 2024-03-29T11:02:18.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.059+0000 7f1004b20640 1 -- start start 2024-03-29T11:02:18.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.059+0000 7f1004b20640 1 librados: setting wanted keys 2024-03-29T11:02:18.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.059+0000 7f1004b20640 1 librados: calling monclient init 2024-03-29T11:02:18.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.059+0000 7f1004b20640 10 monclient: init 2024-03-29T11:02:18.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.060+0000 7f1004b20640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:18.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.060+0000 7f1004b20640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:02:18.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.060+0000 7f1004b20640 10 monclient(hunting): picked mon.noname-c con 0x7f0ff8006e30 addr v1:172.21.15.71:6790/0 2024-03-29T11:02:18.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.060+0000 7f1004b20640 10 monclient(hunting): picked mon.noname-a con 0x7f0ff8035e00 addr v1:172.21.15.71:6789/0 2024-03-29T11:02:18.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.060+0000 7f1004b20640 10 monclient(hunting): picked mon.noname-b con 0x7f0ff80c9670 addr v1:172.21.15.137:6789/0 2024-03-29T11:02:18.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.060+0000 7f1004b20640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff802f440 con 0x7f0ff8035e00 2024-03-29T11:02:18.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.060+0000 7f1004b20640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff8033e20 con 0x7f0ff80c9670 2024-03-29T11:02:18.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.060+0000 7f1004b20640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff802ff40 con 0x7f0ff8006e30 2024-03-29T11:02:18.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.060+0000 7f1004b20640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:18.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.060+0000 7f1004b20640 10 monclient(hunting): authenticate will time out at 894.318264s 2024-03-29T11:02:18.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.061+0000 7f0fff7fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f0ff8006e30 0x7f0ff8032700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:43592/0 (socket says 172.21.15.71:43592) 2024-03-29T11:02:18.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.061+0000 7f0fff7fe640 1 -- 172.21.15.71:0/4074056482 learned_addr learned my addr 172.21.15.71:0/4074056482 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:02:18.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.064+0000 7f0febfff640 10 client.?.objecter ms_handle_connect 0x7f0ff8006e30 2024-03-29T11:02:18.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.064+0000 7f0febfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:18.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.064+0000 7f0febfff640 10 client.?.objecter ms_handle_connect 0x7f0ff80c9670 2024-03-29T11:02:18.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.064+0000 7f0febfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:18.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.064+0000 7f0febfff640 1 -- 172.21.15.71:0/4074056482 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437739072 0 0) 0x7f0ff8033e20 con 0x7f0ff80c9670 2024-03-29T11:02:18.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.064+0000 7f0febfff640 10 monclient(hunting): _negotiate 2024-03-29T11:02:18.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.064+0000 7f0febfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:18.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.064+0000 7f0febfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:18.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.064+0000 7f0febfff640 10 monclient(hunting): my global_id is 4282 2024-03-29T11:02:18.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.064+0000 7f0febfff640 1 -- 172.21.15.71:0/4074056482 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fe4004eb0 con 0x7f0ff80c9670 2024-03-29T11:02:18.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.065+0000 7f0febfff640 1 -- 172.21.15.71:0/4074056482 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755250735 0 0) 0x7f0ff802ff40 con 0x7f0ff8006e30 2024-03-29T11:02:18.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.065+0000 7f0febfff640 10 monclient(hunting): _negotiate 2024-03-29T11:02:18.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.065+0000 7f0febfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:18.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.065+0000 7f0febfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:18.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.065+0000 7f0febfff640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:02:18.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.065+0000 7f0febfff640 1 -- 172.21.15.71:0/4074056482 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ff8033e20 con 0x7f0ff8006e30 2024-03-29T11:02:18.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.065+0000 7f0febfff640 1 -- 172.21.15.71:0/4074056482 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2078497487 0 0) 0x7f0fe4004eb0 con 0x7f0ff80c9670 2024-03-29T11:02:18.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.065+0000 7f0febfff640 1 -- 172.21.15.71:0/4074056482 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ff802ff40 con 0x7f0ff80c9670 2024-03-29T11:02:18.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.065+0000 7f0febfff640 10 client.?.objecter ms_handle_connect 0x7f0ff8035e00 2024-03-29T11:02:18.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.065+0000 7f0febfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:18.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.065+0000 7f0febfff640 1 -- 172.21.15.71:0/4074056482 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f100004fdd0 con 0x7f0ff80c9670 2024-03-29T11:02:18.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.065+0000 7f0febfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:18.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.065+0000 7f0febfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:18.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.065+0000 7f0febfff640 10 monclient(hunting): dump: 2024-03-29T11:02:18.048 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:18.048 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:18.048 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:18.048 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:18.048 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:18.048 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:18.048 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:18.048 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:18.049 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:18.049 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:18.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.065+0000 7f0febfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:18.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.065+0000 7f0febfff640 1 -- 172.21.15.71:0/4074056482 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457348589 0 0) 0x7f0ff802f440 con 0x7f0ff8035e00 2024-03-29T11:02:18.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.065+0000 7f0febfff640 10 monclient(hunting): _negotiate 2024-03-29T11:02:18.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.065+0000 7f0febfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:18.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.065+0000 7f0febfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:18.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.065+0000 7f0febfff640 10 monclient(hunting): my global_id is 4305 2024-03-29T11:02:18.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.065+0000 7f0febfff640 1 -- 172.21.15.71:0/4074056482 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fe4004eb0 con 0x7f0ff8035e00 2024-03-29T11:02:18.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.065+0000 7f0febfff640 1 -- 172.21.15.71:0/4074056482 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2995197966 0 0) 0x7f0ff802ff40 con 0x7f0ff80c9670 2024-03-29T11:02:18.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.065+0000 7f0febfff640 1 -- 172.21.15.71:0/4074056482 >> v1:172.21.15.71:6790/0 conn(0x7f0ff8006e30 legacy=0x7f0ff8032700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:18.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.065+0000 7f0febfff640 1 -- 172.21.15.71:0/4074056482 >> v1:172.21.15.71:6789/0 conn(0x7f0ff8035e00 legacy=0x7f0ff8168ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:18.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.065+0000 7f0febfff640 10 monclient: _finish_hunting 0 2024-03-29T11:02:18.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.065+0000 7f0febfff640 1 monclient: found mon.b 2024-03-29T11:02:18.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.065+0000 7f0febfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:18.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.066+0000 7f0febfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:02:18.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.066+0000 7f0febfff640 1 -- 172.21.15.71:0/4074056482 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff80cd0f0 con 0x7f0ff80c9670 2024-03-29T11:02:18.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.066+0000 7f0febfff640 10 monclient: _finish_auth 0 2024-03-29T11:02:18.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.066+0000 7f0febfff640 10 monclient: _check_auth_tickets 2024-03-29T11:02:18.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.066+0000 7f0febfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:18.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.066+0000 7f1004b20640 5 monclient: authenticate success, global_id 4282 2024-03-29T11:02:18.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.066+0000 7f1004b20640 10 monclient: _renew_subs 2024-03-29T11:02:18.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.066+0000 7f1004b20640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:02:18.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.066+0000 7f1004b20640 1 -- 172.21.15.71:0/4074056482 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff80cd3d0 con 0x7f0ff80c9670 2024-03-29T11:02:18.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.066+0000 7f1004b20640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:18.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.066+0000 7f1004b20640 10 monclient: _renew_subs 2024-03-29T11:02:18.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.066+0000 7f1004b20640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:02:18.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.066+0000 7f1004b20640 1 -- 172.21.15.71:0/4074056482 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff802ed20 con 0x7f0ff80c9670 2024-03-29T11:02:18.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.066+0000 7f1004b20640 1 librados: init done 2024-03-29T11:02:18.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.066+0000 7f0febfff640 1 -- 172.21.15.71:0/4074056482 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f100004f400 con 0x7f0ff80c9670 2024-03-29T11:02:18.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.066+0000 7f0febfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:18.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.066+0000 7f0febfff640 1 -- 172.21.15.71:0/4074056482 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f1000075f10 con 0x7f0ff80c9670 2024-03-29T11:02:18.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.066+0000 7f0febfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:18.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.066+0000 7f0febfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:02:18.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.066+0000 7f0febfff640 10 monclient: dump: 2024-03-29T11:02:18.051 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:18.051 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:18.051 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:18.051 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:18.051 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:18.051 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:18.052 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:18.052 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:18.052 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:18.052 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:18.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.067+0000 7f1004b20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:18.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.067+0000 7f1004b20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:18.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.067+0000 7f1004b20640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:02:18.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.067+0000 7f1004b20640 1 -- 172.21.15.71:0/4074056482 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff8007b90 con 0x7f0ff80c9670 2024-03-29T11:02:18.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.068+0000 7f0ffdffb640 4 set_mon_vals no callback set 2024-03-29T11:02:18.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.070+0000 7f0febfff640 1 -- 172.21.15.71:0/4074056482 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f1000067bf0 con 0x7f0ff80c9670 2024-03-29T11:02:18.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.070+0000 7f0febfff640 1 -- 172.21.15.71:0/4074056482 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f10000fe1c0 con 0x7f0ff80c9670 2024-03-29T11:02:18.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.070+0000 7f0febfff640 10 client.4282.objecter ms_dispatch 0x7f1000000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:02:18.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.070+0000 7f0febfff640 3 client.4282.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:02:18.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.070+0000 7f0febfff640 3 client.4282.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:02:18.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.071+0000 7f0febfff640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-29T11:02:18.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.071+0000 7f0fff7fe640 10 monclient: get_auth_request con 0x7f0fe406dd00 auth_method 0 2024-03-29T11:02:18.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.071+0000 7f0febfff640 10 client.4282.objecter ms_handle_connect 0x7f0fe406dd00 2024-03-29T11:02:18.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.076+0000 7f0febfff640 1 -- 172.21.15.71:0/4074056482 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f10000c9aa0 con 0x7f0ff80c9670 2024-03-29T11:02:18.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.076+0000 7f0febfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:18.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.076+0000 7f0febfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:18.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.395+0000 7f1004b20640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:18.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.395+0000 7f1004b20640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:18.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.395+0000 7f1004b20640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:02:18.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.395+0000 7f1004b20640 1 -- 172.21.15.71:0/4074056482 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ff8005120 con 0x7f0ff80c9670 2024-03-29T11:02:18.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.399+0000 7f0febfff640 1 -- 172.21.15.71:0/4074056482 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f10000cb9d0 con 0x7f0ff80c9670 2024-03-29T11:02:18.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.399+0000 7f0febfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:18.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.399+0000 7f0febfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:18.379 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:02:18.380 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:18.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.403+0000 7f1004b20640 10 librados: watch_flush enter 2024-03-29T11:02:18.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.405+0000 7f1004b20640 10 librados: watch_flush exit 2024-03-29T11:02:18.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.405+0000 7f1004b20640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-29T11:02:18.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.405+0000 7f1004b20640 10 client.4282.objecter successfully canceled tick 2024-03-29T11:02:18.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.405+0000 7f1004b20640 1 -- 172.21.15.71:0/4074056482 >> v1:172.21.15.137:6816/973332665 conn(0x7f0fe406dd00 legacy=0x7f0fe40701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:18.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.405+0000 7f1004b20640 10 monclient: shutdown 2024-03-29T11:02:18.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.405+0000 7f1004b20640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:18.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.405+0000 7f1004b20640 1 -- 172.21.15.71:0/4074056482 >> v1:172.21.15.137:6789/0 conn(0x7f0ff80c9670 legacy=0x7f0ff816c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:18.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.405+0000 7f1004b20640 1 -- 172.21.15.71:0/4074056482 shutdown_connections 2024-03-29T11:02:18.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.405+0000 7f1004b20640 1 -- 172.21.15.71:0/4074056482 >> 172.21.15.71:0/4074056482 conn(0x7f0ff8005c00 msgr2=0x7f0ff80c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:18.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.405+0000 7f1004b20640 1 -- 172.21.15.71:0/4074056482 shutdown_connections 2024-03-29T11:02:18.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.405+0000 7f1004b20640 1 -- 172.21.15.71:0/4074056482 wait complete. 2024-03-29T11:02:18.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:18.405+0000 7f1004b20640 1 librados: shutdown 2024-03-29T11:02:18.403 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:25.211 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:25.360 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.379+0000 7f96baa74640 10 monclient: get_monmap_and_config 2024-03-29T11:02:25.360 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.379+0000 7f96baa74640 10 monclient: build_initial_monmap 2024-03-29T11:02:25.360 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.379+0000 7f96baa74640 10 monclient: monmap: 2024-03-29T11:02:25.360 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:02:25.360 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:25.360 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:02:25.380607+0000 2024-03-29T11:02:25.361 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:02:25.380607+0000 2024-03-29T11:02:25.361 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:25.361 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:25.361 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:02:25.361 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:02:25.361 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:02:25.361 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:25.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.381+0000 7f96baa74640 1 Processor -- start 2024-03-29T11:02:25.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.381+0000 7f96baa74640 1 -- start start 2024-03-29T11:02:25.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.381+0000 7f96baa74640 10 monclient: init 2024-03-29T11:02:25.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.382+0000 7f96baa74640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:25.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.382+0000 7f96baa74640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:02:25.365 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.384+0000 7f96baa74640 10 monclient(hunting): picked mon.noname-a con 0x7f96ac006890 addr v1:172.21.15.71:6789/0 2024-03-29T11:02:25.365 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.384+0000 7f96baa74640 10 monclient(hunting): picked mon.noname-b con 0x7f96ac006c70 addr v1:172.21.15.71:6790/0 2024-03-29T11:02:25.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.384+0000 7f96baa74640 10 monclient(hunting): picked mon.noname-c con 0x7f96ac02b4c0 addr v1:172.21.15.137:6789/0 2024-03-29T11:02:25.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.384+0000 7f96baa74640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ac02c830 con 0x7f96ac006890 2024-03-29T11:02:25.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.384+0000 7f96baa74640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ac02afa0 con 0x7f96ac02b4c0 2024-03-29T11:02:25.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.384+0000 7f96baa74640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ac02aba0 con 0x7f96ac006c70 2024-03-29T11:02:25.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.384+0000 7f96baa74640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:25.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.384+0000 7f96baa74640 10 monclient(hunting): authenticate will time out at 901.642396s 2024-03-29T11:02:25.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.384+0000 7f96b9a72640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f96ac006890 0x7f96ac0c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:58691/0 (socket says 172.21.15.71:58691) 2024-03-29T11:02:25.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.384+0000 7f96b9a72640 1 -- 172.21.15.71:0/2142913992 learned_addr learned my addr 172.21.15.71:0/2142913992 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:02:25.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.385+0000 7f96b8a70640 1 -- 172.21.15.71:0/2142913992 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2923414845 0 0) 0x7f96ac02c830 con 0x7f96ac006890 2024-03-29T11:02:25.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.385+0000 7f96b8a70640 10 monclient(hunting): _negotiate 2024-03-29T11:02:25.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.385+0000 7f96b8a70640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:25.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.385+0000 7f96b8a70640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:25.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.385+0000 7f96b8a70640 10 monclient(hunting): my global_id is 4308 2024-03-29T11:02:25.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.385+0000 7f96b8a70640 1 -- 172.21.15.71:0/2142913992 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96a0003520 con 0x7f96ac006890 2024-03-29T11:02:25.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.385+0000 7f96b8a70640 1 -- 172.21.15.71:0/2142913992 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 247476875 0 0) 0x7f96a0003520 con 0x7f96ac006890 2024-03-29T11:02:25.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.385+0000 7f96b8a70640 1 -- 172.21.15.71:0/2142913992 >> v1:172.21.15.71:6790/0 conn(0x7f96ac006c70 legacy=0x7f96ac0c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:25.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.385+0000 7f96b8a70640 1 -- 172.21.15.71:0/2142913992 >> v1:172.21.15.137:6789/0 conn(0x7f96ac02b4c0 legacy=0x7f96ac0cf970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:02:25.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.387+0000 7f96b8a70640 10 monclient: _finish_hunting 0 2024-03-29T11:02:25.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.387+0000 7f96b8a70640 1 monclient: found mon.noname-a 2024-03-29T11:02:25.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.387+0000 7f96b8a70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:25.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.387+0000 7f96b8a70640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:02:25.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.387+0000 7f96b8a70640 1 -- 172.21.15.71:0/2142913992 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ac02ad30 con 0x7f96ac006890 2024-03-29T11:02:25.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.387+0000 7f96b8a70640 10 monclient: _finish_auth 0 2024-03-29T11:02:25.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.387+0000 7f96b8a70640 10 monclient: _check_auth_tickets 2024-03-29T11:02:25.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.387+0000 7f96b8a70640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:25.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.387+0000 7f96baa74640 5 monclient: authenticate success, global_id 4308 2024-03-29T11:02:25.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.387+0000 7f96baa74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:25.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.387+0000 7f96b8a70640 1 -- 172.21.15.71:0/2142913992 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f96b404f730 con 0x7f96ac006890 2024-03-29T11:02:25.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.387+0000 7f96b8a70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:25.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.387+0000 7f96b8a70640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:02:25.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.387+0000 7f96b8a70640 10 monclient: dump: 2024-03-29T11:02:25.373 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:25.373 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:25.373 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:25.373 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:25.373 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:25.373 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:25.373 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:25.373 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:25.374 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:25.374 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:25.374 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.387+0000 7f96baa74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:25.374 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.387+0000 7f96b8a70640 1 -- 172.21.15.71:0/2142913992 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96b40008d0 con 0x7f96ac006890 2024-03-29T11:02:25.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.387+0000 7f96b8a70640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:25.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.387+0000 7f96baa74640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:25.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.387+0000 7f96baa74640 4 set_mon_vals no callback set 2024-03-29T11:02:25.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.388+0000 7f96b8a70640 1 -- 172.21.15.71:0/2142913992 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f96b40758b0 con 0x7f96ac006890 2024-03-29T11:02:25.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.388+0000 7f96b8a70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:25.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.388+0000 7f96b8a70640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:25.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.388+0000 7f96b8a70640 10 monclient: dump: 2024-03-29T11:02:25.375 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:25.375 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:25.375 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:25.375 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:25.375 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:25.375 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:25.375 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:25.375 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:25.376 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:25.376 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:25.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.388+0000 7f96baa74640 10 monclient: shutdown 2024-03-29T11:02:25.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.388+0000 7f96baa74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:25.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.388+0000 7f96baa74640 1 -- 172.21.15.71:0/2142913992 >> v1:172.21.15.71:6789/0 conn(0x7f96ac006890 legacy=0x7f96ac0c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:25.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.388+0000 7f96baa74640 1 -- 172.21.15.71:0/2142913992 shutdown_connections 2024-03-29T11:02:25.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.388+0000 7f96baa74640 1 -- 172.21.15.71:0/2142913992 >> 172.21.15.71:0/2142913992 conn(0x7f96ac0058f0 msgr2=0x7f96ac005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:25.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.390+0000 7f96baa74640 1 -- 172.21.15.71:0/2142913992 shutdown_connections 2024-03-29T11:02:25.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.390+0000 7f96baa74640 1 -- 172.21.15.71:0/2142913992 wait complete. 2024-03-29T11:02:25.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.391+0000 7f96baa74640 10 monclient: build_initial_monmap 2024-03-29T11:02:25.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.391+0000 7f96baa74640 10 monclient: monmap: 2024-03-29T11:02:25.376 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:02:25.376 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:25.376 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:02:25.376 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:02:25.377 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:25.377 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:25.377 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:02:25.377 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:02:25.377 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:02:25.377 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:25.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.391+0000 7f96baa74640 1 librados: starting msgr at 2024-03-29T11:02:25.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.391+0000 7f96baa74640 1 librados: starting objecter 2024-03-29T11:02:25.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.391+0000 7f96baa74640 1 Processor -- start 2024-03-29T11:02:25.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.392+0000 7f96baa74640 1 -- start start 2024-03-29T11:02:25.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.392+0000 7f96baa74640 1 librados: setting wanted keys 2024-03-29T11:02:25.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.392+0000 7f96baa74640 1 librados: calling monclient init 2024-03-29T11:02:25.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.392+0000 7f96baa74640 10 monclient: init 2024-03-29T11:02:25.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.392+0000 7f96baa74640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:25.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.392+0000 7f96baa74640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:02:25.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.396+0000 7f96baa74640 10 monclient(hunting): picked mon.noname-a con 0x7f96ac006890 addr v1:172.21.15.71:6789/0 2024-03-29T11:02:25.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.396+0000 7f96baa74640 10 monclient(hunting): picked mon.noname-b con 0x7f96ac02b4c0 addr v1:172.21.15.137:6789/0 2024-03-29T11:02:25.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.396+0000 7f96baa74640 10 monclient(hunting): picked mon.noname-c con 0x7f96ac0c40a0 addr v1:172.21.15.71:6790/0 2024-03-29T11:02:25.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.396+0000 7f96baa74640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ac030470 con 0x7f96ac006890 2024-03-29T11:02:25.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.396+0000 7f96baa74640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ac030750 con 0x7f96ac02b4c0 2024-03-29T11:02:25.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.396+0000 7f96baa74640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ac030ff0 con 0x7f96ac0c40a0 2024-03-29T11:02:25.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.396+0000 7f96baa74640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:25.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.396+0000 7f96baa74640 10 monclient(hunting): authenticate will time out at 901.654839s 2024-03-29T11:02:25.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.397+0000 7f96b9a72640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f96ac006890 0x7f96ac02e470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:58877/0 (socket says 172.21.15.71:58877) 2024-03-29T11:02:25.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.397+0000 7f96b9a72640 1 -- 172.21.15.71:0/3876963113 learned_addr learned my addr 172.21.15.71:0/3876963113 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:02:25.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.397+0000 7f96aa7fc640 10 client.?.objecter ms_handle_connect 0x7f96ac006890 2024-03-29T11:02:25.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.397+0000 7f96aa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:25.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.397+0000 7f96aa7fc640 1 -- 172.21.15.71:0/3876963113 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316701446 0 0) 0x7f96ac030470 con 0x7f96ac006890 2024-03-29T11:02:25.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.397+0000 7f96aa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:02:25.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.397+0000 7f96aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:25.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.397+0000 7f96aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:25.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.397+0000 7f96aa7fc640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:02:25.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.397+0000 7f96aa7fc640 1 -- 172.21.15.71:0/3876963113 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9698004eb0 con 0x7f96ac006890 2024-03-29T11:02:25.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.398+0000 7f96aa7fc640 1 -- 172.21.15.71:0/3876963113 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1938530171 0 0) 0x7f9698004eb0 con 0x7f96ac006890 2024-03-29T11:02:25.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.398+0000 7f96aa7fc640 1 -- 172.21.15.71:0/3876963113 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96ac030470 con 0x7f96ac006890 2024-03-29T11:02:25.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.398+0000 7f96aa7fc640 1 -- 172.21.15.71:0/3876963113 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f96b404f400 con 0x7f96ac006890 2024-03-29T11:02:25.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.398+0000 7f96aa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:25.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.398+0000 7f96aa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:02:25.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.398+0000 7f96aa7fc640 10 monclient(hunting): dump: 2024-03-29T11:02:25.381 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:25.381 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:25.381 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:25.381 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:25.381 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:25.381 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:25.381 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:25.381 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:25.382 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:25.382 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:25.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.398+0000 7f96aa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:25.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.401+0000 7f96aa7fc640 1 -- 172.21.15.71:0/3876963113 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2726323349 0 0) 0x7f96ac030470 con 0x7f96ac006890 2024-03-29T11:02:25.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.401+0000 7f96aa7fc640 1 -- 172.21.15.71:0/3876963113 >> v1:172.21.15.71:6790/0 conn(0x7f96ac0c40a0 legacy=0x7f96ac174100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:25.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.401+0000 7f96aa7fc640 1 -- 172.21.15.71:0/3876963113 >> v1:172.21.15.137:6789/0 conn(0x7f96ac02b4c0 legacy=0x7f96ac031510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:25.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.401+0000 7f96aa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:02:25.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.401+0000 7f96aa7fc640 1 monclient: found mon.a 2024-03-29T11:02:25.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.401+0000 7f96aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:25.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.401+0000 7f96aa7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:02:25.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.401+0000 7f96aa7fc640 1 -- 172.21.15.71:0/3876963113 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ac16cbc0 con 0x7f96ac006890 2024-03-29T11:02:25.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.401+0000 7f96aa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:02:25.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.401+0000 7f96aa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:02:25.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.401+0000 7f96aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:25.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.401+0000 7f96aa7fc640 10 client.?.objecter ms_handle_connect 0x7f96ac02b4c0 2024-03-29T11:02:25.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.401+0000 7f96aa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:25.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.401+0000 7f96aa7fc640 1 -- 172.21.15.71:0/3876963113 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96b404f730 con 0x7f96ac006890 2024-03-29T11:02:25.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.401+0000 7f96aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:25.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.401+0000 7f96aa7fc640 1 -- 172.21.15.71:0/3876963113 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f96b4067d90 con 0x7f96ac006890 2024-03-29T11:02:25.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.401+0000 7f96aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:25.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.401+0000 7f96aa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:25.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.401+0000 7f96aa7fc640 10 monclient: dump: 2024-03-29T11:02:25.384 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:25.384 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:25.384 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:25.384 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:25.384 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:25.384 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:25.384 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:25.384 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:25.384 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:25.384 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:25.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.401+0000 7f96ab7fe640 4 set_mon_vals no callback set 2024-03-29T11:02:25.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.402+0000 7f96baa74640 5 monclient: authenticate success, global_id 4311 2024-03-29T11:02:25.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.402+0000 7f96baa74640 10 monclient: _renew_subs 2024-03-29T11:02:25.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.402+0000 7f96baa74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:02:25.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.402+0000 7f96baa74640 1 -- 172.21.15.71:0/3876963113 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96ac16ce70 con 0x7f96ac006890 2024-03-29T11:02:25.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.402+0000 7f96baa74640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:25.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.402+0000 7f96baa74640 10 monclient: _renew_subs 2024-03-29T11:02:25.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.402+0000 7f96baa74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:02:25.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.402+0000 7f96baa74640 1 -- 172.21.15.71:0/3876963113 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96ac175a00 con 0x7f96ac006890 2024-03-29T11:02:25.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.402+0000 7f96baa74640 1 librados: init done 2024-03-29T11:02:25.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.404+0000 7f96baa74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:25.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.404+0000 7f96baa74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:25.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.404+0000 7f96baa74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:02:25.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.404+0000 7f96baa74640 1 -- 172.21.15.71:0/3876963113 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96ac030e50 con 0x7f96ac006890 2024-03-29T11:02:25.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.405+0000 7f96aa7fc640 1 -- 172.21.15.71:0/3876963113 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f96b4067ef0 con 0x7f96ac006890 2024-03-29T11:02:25.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.406+0000 7f96aa7fc640 1 -- 172.21.15.71:0/3876963113 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f96b40ff630 con 0x7f96ac006890 2024-03-29T11:02:25.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.406+0000 7f96aa7fc640 10 client.4311.objecter ms_dispatch 0x7f96b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:02:25.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.406+0000 7f96aa7fc640 3 client.4311.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:02:25.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.406+0000 7f96aa7fc640 3 client.4311.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:02:25.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.406+0000 7f96aa7fc640 20 client.4311.objecter dump_active .. 0 homeless 2024-03-29T11:02:25.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.407+0000 7f96b9271640 10 monclient: get_auth_request con 0x7f969806dbf0 auth_method 0 2024-03-29T11:02:25.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.414+0000 7f96aa7fc640 10 client.4311.objecter ms_handle_connect 0x7f969806dbf0 2024-03-29T11:02:25.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.414+0000 7f96aa7fc640 1 -- 172.21.15.71:0/3876963113 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f96ac030e50 con 0x7f96ac006890 2024-03-29T11:02:25.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.414+0000 7f96aa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:25.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.414+0000 7f96aa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:25.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.747+0000 7f96baa74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:25.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.747+0000 7f96baa74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:25.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.747+0000 7f96baa74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:02:25.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.747+0000 7f96baa74640 1 -- 172.21.15.71:0/3876963113 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96ac030e50 con 0x7f96ac006890 2024-03-29T11:02:25.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.749+0000 7f96aa7fc640 1 -- 172.21.15.71:0/3876963113 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f96ac030e50 con 0x7f96ac006890 2024-03-29T11:02:25.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.749+0000 7f96aa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:25.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.749+0000 7f96aa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:25.730 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:02:25.730 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:25.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.753+0000 7f96baa74640 10 librados: watch_flush enter 2024-03-29T11:02:25.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.753+0000 7f96baa74640 10 librados: watch_flush exit 2024-03-29T11:02:25.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.753+0000 7f96baa74640 20 client.4311.objecter shutdown clearing up homeless session... 2024-03-29T11:02:25.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.753+0000 7f96baa74640 10 client.4311.objecter successfully canceled tick 2024-03-29T11:02:25.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.753+0000 7f96baa74640 1 -- 172.21.15.71:0/3876963113 >> v1:172.21.15.137:6816/973332665 conn(0x7f969806dbf0 legacy=0x7f96980700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:25.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.753+0000 7f96baa74640 10 monclient: shutdown 2024-03-29T11:02:25.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.753+0000 7f96baa74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:25.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.753+0000 7f96baa74640 1 -- 172.21.15.71:0/3876963113 >> v1:172.21.15.71:6789/0 conn(0x7f96ac006890 legacy=0x7f96ac02e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:25.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.753+0000 7f96baa74640 1 -- 172.21.15.71:0/3876963113 shutdown_connections 2024-03-29T11:02:25.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.753+0000 7f96baa74640 1 -- 172.21.15.71:0/3876963113 >> 172.21.15.71:0/3876963113 conn(0x7f96ac0058f0 msgr2=0x7f96ac038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:25.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.754+0000 7f96baa74640 1 -- 172.21.15.71:0/3876963113 shutdown_connections 2024-03-29T11:02:25.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.754+0000 7f96baa74640 1 -- 172.21.15.71:0/3876963113 wait complete. 2024-03-29T11:02:25.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:25.755+0000 7f96baa74640 1 librados: shutdown 2024-03-29T11:02:25.754 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:32.562 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:32.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.735+0000 7fb23caa8640 10 monclient: get_monmap_and_config 2024-03-29T11:02:32.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.735+0000 7fb23caa8640 10 monclient: build_initial_monmap 2024-03-29T11:02:32.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.735+0000 7fb23caa8640 10 monclient: monmap: 2024-03-29T11:02:32.715 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:02:32.715 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:32.715 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:02:32.736320+0000 2024-03-29T11:02:32.715 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:02:32.736320+0000 2024-03-29T11:02:32.715 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:32.715 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:32.716 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:02:32.716 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:02:32.716 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:02:32.716 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:32.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.736+0000 7fb23caa8640 1 Processor -- start 2024-03-29T11:02:32.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.737+0000 7fb23caa8640 1 -- start start 2024-03-29T11:02:32.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.737+0000 7fb23caa8640 10 monclient: init 2024-03-29T11:02:32.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.738+0000 7fb23caa8640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:32.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.738+0000 7fb23caa8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:02:32.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.738+0000 7fb23caa8640 10 monclient(hunting): picked mon.noname-c con 0x7fb2380b1f60 addr v1:172.21.15.137:6789/0 2024-03-29T11:02:32.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.738+0000 7fb23caa8640 10 monclient(hunting): picked mon.noname-a con 0x7fb238071eb0 addr v1:172.21.15.71:6789/0 2024-03-29T11:02:32.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.738+0000 7fb23caa8640 10 monclient(hunting): picked mon.noname-b con 0x7fb238143d10 addr v1:172.21.15.71:6790/0 2024-03-29T11:02:32.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.738+0000 7fb23caa8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb23806ef80 con 0x7fb238071eb0 2024-03-29T11:02:32.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.738+0000 7fb23caa8640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb23806f260 con 0x7fb2380b1f60 2024-03-29T11:02:32.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.738+0000 7fb23caa8640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2380b40c0 con 0x7fb238143d10 2024-03-29T11:02:32.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.738+0000 7fb23caa8640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:32.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.738+0000 7fb23caa8640 10 monclient(hunting): authenticate will time out at 908.996256s 2024-03-29T11:02:32.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.742+0000 7fb235d9b640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fb238071eb0 0x7fb238146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:35611/0 (socket says 172.21.15.71:35611) 2024-03-29T11:02:32.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.742+0000 7fb235d9b640 1 -- 172.21.15.71:0/2845160404 learned_addr learned my addr 172.21.15.71:0/2845160404 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:02:32.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.744+0000 7fb23559a640 1 -- 172.21.15.71:0/2845160404 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 326174212 0 0) 0x7fb23806ef80 con 0x7fb238071eb0 2024-03-29T11:02:32.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.744+0000 7fb23559a640 10 monclient(hunting): _negotiate 2024-03-29T11:02:32.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.744+0000 7fb23559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:32.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.744+0000 7fb23559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:32.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.744+0000 7fb23559a640 10 monclient(hunting): my global_id is 4314 2024-03-29T11:02:32.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.744+0000 7fb23559a640 1 -- 172.21.15.71:0/2845160404 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb220003520 con 0x7fb238071eb0 2024-03-29T11:02:32.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.744+0000 7fb23559a640 1 -- 172.21.15.71:0/2845160404 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1117570864 0 0) 0x7fb220003520 con 0x7fb238071eb0 2024-03-29T11:02:32.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.744+0000 7fb23559a640 1 -- 172.21.15.71:0/2845160404 >> v1:172.21.15.71:6790/0 conn(0x7fb238143d10 legacy=0x7fb238146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:02:32.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.745+0000 7fb23559a640 1 -- 172.21.15.71:0/2845160404 >> v1:172.21.15.137:6789/0 conn(0x7fb2380b1f60 legacy=0x7fb238143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:02:32.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.745+0000 7fb23559a640 10 monclient: _finish_hunting 0 2024-03-29T11:02:32.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.745+0000 7fb23559a640 1 monclient: found mon.noname-a 2024-03-29T11:02:32.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.745+0000 7fb23559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:32.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.745+0000 7fb23559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:02:32.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.745+0000 7fb23559a640 1 -- 172.21.15.71:0/2845160404 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb238155d70 con 0x7fb238071eb0 2024-03-29T11:02:32.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.745+0000 7fb23559a640 10 monclient: _finish_auth 0 2024-03-29T11:02:32.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.745+0000 7fb23559a640 10 monclient: _check_auth_tickets 2024-03-29T11:02:32.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.745+0000 7fb23559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:32.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.745+0000 7fb23559a640 1 -- 172.21.15.71:0/2845160404 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb230001a10 con 0x7fb238071eb0 2024-03-29T11:02:32.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.745+0000 7fb23559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:32.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.745+0000 7fb23559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:02:32.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.745+0000 7fb23559a640 10 monclient: dump: 2024-03-29T11:02:32.730 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:32.730 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:32.730 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:32.730 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:32.730 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:32.730 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:32.730 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:32.730 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:32.730 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:32.731 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:32.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.745+0000 7fb23559a640 1 -- 172.21.15.71:0/2845160404 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb230001be0 con 0x7fb238071eb0 2024-03-29T11:02:32.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.745+0000 7fb23559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:32.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.745+0000 7fb23559a640 1 -- 172.21.15.71:0/2845160404 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb230002030 con 0x7fb238071eb0 2024-03-29T11:02:32.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.746+0000 7fb23559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:32.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.746+0000 7fb23559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:32.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.746+0000 7fb23559a640 10 monclient: dump: 2024-03-29T11:02:32.731 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:32.731 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:32.731 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:32.731 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:32.731 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:32.731 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:32.731 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:32.731 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:32.732 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:32.732 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:32.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.746+0000 7fb23caa8640 5 monclient: authenticate success, global_id 4314 2024-03-29T11:02:32.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.746+0000 7fb23caa8640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:32.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.746+0000 7fb23caa8640 4 set_mon_vals no callback set 2024-03-29T11:02:32.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.746+0000 7fb23caa8640 10 monclient: shutdown 2024-03-29T11:02:32.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.746+0000 7fb23caa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:32.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.746+0000 7fb23caa8640 1 -- 172.21.15.71:0/2845160404 >> v1:172.21.15.71:6789/0 conn(0x7fb238071eb0 legacy=0x7fb238146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:32.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.747+0000 7fb23caa8640 1 -- 172.21.15.71:0/2845160404 shutdown_connections 2024-03-29T11:02:32.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.747+0000 7fb23caa8640 1 -- 172.21.15.71:0/2845160404 >> 172.21.15.71:0/2845160404 conn(0x7fb238066a00 msgr2=0x7fb238066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:32.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.747+0000 7fb23caa8640 1 -- 172.21.15.71:0/2845160404 shutdown_connections 2024-03-29T11:02:32.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.747+0000 7fb23caa8640 1 -- 172.21.15.71:0/2845160404 wait complete. 2024-03-29T11:02:32.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.748+0000 7fb23caa8640 10 monclient: build_initial_monmap 2024-03-29T11:02:32.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.748+0000 7fb23caa8640 10 monclient: monmap: 2024-03-29T11:02:32.732 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:02:32.733 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:32.733 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:02:32.733 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:02:32.733 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:32.733 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:32.733 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:02:32.733 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:02:32.733 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:02:32.733 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:32.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.748+0000 7fb23caa8640 1 librados: starting msgr at 2024-03-29T11:02:32.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.748+0000 7fb23caa8640 1 librados: starting objecter 2024-03-29T11:02:32.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.748+0000 7fb23caa8640 1 Processor -- start 2024-03-29T11:02:32.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.748+0000 7fb23caa8640 1 -- start start 2024-03-29T11:02:32.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.748+0000 7fb23caa8640 1 librados: setting wanted keys 2024-03-29T11:02:32.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.748+0000 7fb23caa8640 1 librados: calling monclient init 2024-03-29T11:02:32.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.748+0000 7fb23caa8640 10 monclient: init 2024-03-29T11:02:32.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.749+0000 7fb23caa8640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:32.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.749+0000 7fb23caa8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:02:32.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.749+0000 7fb23caa8640 10 monclient(hunting): picked mon.noname-a con 0x7fb2380b1f60 addr v1:172.21.15.71:6789/0 2024-03-29T11:02:32.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.753+0000 7fb23caa8640 10 monclient(hunting): picked mon.noname-b con 0x7fb238143d10 addr v1:172.21.15.137:6789/0 2024-03-29T11:02:32.743 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.756+0000 7fb235d9b640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fb238143d10 0x7fb2380a9aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:51625/0 (socket says 172.21.15.71:51625) 2024-03-29T11:02:32.743 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.756+0000 7fb23caa8640 10 monclient(hunting): picked mon.noname-c con 0x7fb2380ab250 addr v1:172.21.15.71:6790/0 2024-03-29T11:02:32.743 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.756+0000 7fb23caa8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2380a9610 con 0x7fb2380b1f60 2024-03-29T11:02:32.743 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.756+0000 7fb23caa8640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2380aac90 con 0x7fb238143d10 2024-03-29T11:02:32.743 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.756+0000 7fb23caa8640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2380ab7d0 con 0x7fb2380ab250 2024-03-29T11:02:32.743 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.756+0000 7fb23caa8640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:32.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.756+0000 7fb23caa8640 10 monclient(hunting): authenticate will time out at 909.014898s 2024-03-29T11:02:32.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.756+0000 7fb235d9b640 1 -- 172.21.15.71:0/3208673296 learned_addr learned my addr 172.21.15.71:0/3208673296 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:02:32.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.757+0000 7fb226ffd640 10 client.?.objecter ms_handle_connect 0x7fb2380ab250 2024-03-29T11:02:32.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.757+0000 7fb226ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:32.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.757+0000 7fb226ffd640 1 -- 172.21.15.71:0/3208673296 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2144526682 0 0) 0x7fb2380ab7d0 con 0x7fb2380ab250 2024-03-29T11:02:32.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.757+0000 7fb226ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:32.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.757+0000 7fb226ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:32.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.757+0000 7fb226ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:32.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.757+0000 7fb226ffd640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:02:32.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.757+0000 7fb226ffd640 1 -- 172.21.15.71:0/3208673296 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb218004eb0 con 0x7fb2380ab250 2024-03-29T11:02:32.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.758+0000 7fb226ffd640 1 -- 172.21.15.71:0/3208673296 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2489378478 0 0) 0x7fb218004eb0 con 0x7fb2380ab250 2024-03-29T11:02:32.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.758+0000 7fb226ffd640 1 -- 172.21.15.71:0/3208673296 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2380ab7d0 con 0x7fb2380ab250 2024-03-29T11:02:32.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.758+0000 7fb226ffd640 1 -- 172.21.15.71:0/3208673296 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb22c001750 con 0x7fb2380ab250 2024-03-29T11:02:32.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.758+0000 7fb226ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:32.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.758+0000 7fb226ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:32.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.758+0000 7fb226ffd640 10 monclient(hunting): dump: 2024-03-29T11:02:32.745 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:32.745 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:32.745 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:32.745 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:32.745 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:32.745 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:32.745 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:32.745 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:32.745 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:32.745 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:32.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.758+0000 7fb226ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:32.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.758+0000 7fb226ffd640 1 -- 172.21.15.71:0/3208673296 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2772958116 0 0) 0x7fb2380ab7d0 con 0x7fb2380ab250 2024-03-29T11:02:32.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.758+0000 7fb226ffd640 1 -- 172.21.15.71:0/3208673296 >> v1:172.21.15.137:6789/0 conn(0x7fb238143d10 legacy=0x7fb2380a9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:32.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.758+0000 7fb226ffd640 1 -- 172.21.15.71:0/3208673296 >> v1:172.21.15.71:6789/0 conn(0x7fb2380b1f60 legacy=0x7fb23813fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:02:32.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.759+0000 7fb226ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:02:32.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.759+0000 7fb226ffd640 1 monclient: found mon.c 2024-03-29T11:02:32.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.759+0000 7fb226ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:32.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.759+0000 7fb226ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:02:32.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.759+0000 7fb226ffd640 1 -- 172.21.15.71:0/3208673296 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2380ab960 con 0x7fb2380ab250 2024-03-29T11:02:32.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.759+0000 7fb226ffd640 10 monclient: _finish_auth 0 2024-03-29T11:02:32.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.759+0000 7fb226ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:02:32.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.759+0000 7fb226ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:32.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.759+0000 7fb23caa8640 5 monclient: authenticate success, global_id 4301 2024-03-29T11:02:32.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.759+0000 7fb23caa8640 10 monclient: _renew_subs 2024-03-29T11:02:32.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.759+0000 7fb23caa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:02:32.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.759+0000 7fb23caa8640 1 -- 172.21.15.71:0/3208673296 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb238140460 con 0x7fb2380ab250 2024-03-29T11:02:32.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.759+0000 7fb23caa8640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:32.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.759+0000 7fb23caa8640 10 monclient: _renew_subs 2024-03-29T11:02:32.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.759+0000 7fb23caa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:02:32.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.759+0000 7fb23caa8640 1 -- 172.21.15.71:0/3208673296 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb238154a30 con 0x7fb2380ab250 2024-03-29T11:02:32.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.759+0000 7fb23caa8640 1 librados: init done 2024-03-29T11:02:32.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.760+0000 7fb226ffd640 1 -- 172.21.15.71:0/3208673296 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb22c0018b0 con 0x7fb2380ab250 2024-03-29T11:02:32.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.760+0000 7fb226ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:32.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.760+0000 7fb234d99640 4 set_mon_vals no callback set 2024-03-29T11:02:32.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.761+0000 7fb23caa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:32.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.761+0000 7fb226ffd640 1 -- 172.21.15.71:0/3208673296 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb22c003790 con 0x7fb2380ab250 2024-03-29T11:02:32.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.761+0000 7fb226ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:32.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.761+0000 7fb226ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:32.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.761+0000 7fb226ffd640 10 monclient: dump: 2024-03-29T11:02:32.748 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:32.748 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:32.748 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:32.748 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:32.748 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:32.748 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:32.748 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:32.748 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:32.748 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:32.748 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:32.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.761+0000 7fb23caa8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:32.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.761+0000 7fb23caa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:02:32.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.761+0000 7fb23caa8640 1 -- 172.21.15.71:0/3208673296 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb21c005080 con 0x7fb2380ab250 2024-03-29T11:02:32.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.761+0000 7fb226ffd640 1 -- 172.21.15.71:0/3208673296 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7fb22c003970 con 0x7fb2380ab250 2024-03-29T11:02:32.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.762+0000 7fb226ffd640 1 -- 172.21.15.71:0/3208673296 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7fb22c0837b0 con 0x7fb2380ab250 2024-03-29T11:02:32.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.762+0000 7fb226ffd640 10 client.4301.objecter ms_dispatch 0x7fb238000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:02:32.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.762+0000 7fb226ffd640 3 client.4301.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:02:32.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.762+0000 7fb226ffd640 3 client.4301.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:02:32.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.762+0000 7fb226ffd640 20 client.4301.objecter dump_active .. 0 homeless 2024-03-29T11:02:32.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.765+0000 7fb23659c640 10 monclient: get_auth_request con 0x7fb21806dc60 auth_method 0 2024-03-29T11:02:32.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.765+0000 7fb226ffd640 10 client.4301.objecter ms_handle_connect 0x7fb21806dc60 2024-03-29T11:02:32.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.772+0000 7fb226ffd640 1 -- 172.21.15.71:0/3208673296 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb22c04eec0 con 0x7fb2380ab250 2024-03-29T11:02:32.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.772+0000 7fb226ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:32.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:32.772+0000 7fb226ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:33.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:33.101+0000 7fb23caa8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:33.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:33.101+0000 7fb23caa8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:33.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:33.101+0000 7fb23caa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:02:33.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:33.101+0000 7fb23caa8640 1 -- 172.21.15.71:0/3208673296 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb21c005330 con 0x7fb2380ab250 2024-03-29T11:02:33.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:33.102+0000 7fb226ffd640 1 -- 172.21.15.71:0/3208673296 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7fb22c050fc0 con 0x7fb2380ab250 2024-03-29T11:02:33.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:33.102+0000 7fb226ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:33.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:33.102+0000 7fb226ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:33.083 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:02:33.083 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:33.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:33.106+0000 7fb23caa8640 10 librados: watch_flush enter 2024-03-29T11:02:33.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:33.107+0000 7fb23caa8640 10 librados: watch_flush exit 2024-03-29T11:02:33.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:33.107+0000 7fb23caa8640 20 client.4301.objecter shutdown clearing up homeless session... 2024-03-29T11:02:33.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:33.107+0000 7fb23caa8640 10 client.4301.objecter successfully canceled tick 2024-03-29T11:02:33.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:33.107+0000 7fb23caa8640 1 -- 172.21.15.71:0/3208673296 >> v1:172.21.15.137:6816/973332665 conn(0x7fb21806dc60 legacy=0x7fb218070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:33.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:33.107+0000 7fb23caa8640 10 monclient: shutdown 2024-03-29T11:02:33.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:33.107+0000 7fb23caa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:33.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:33.107+0000 7fb23caa8640 1 -- 172.21.15.71:0/3208673296 >> v1:172.21.15.71:6790/0 conn(0x7fb2380ab250 legacy=0x7fb2380aa1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:33.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:33.107+0000 7fb23caa8640 1 -- 172.21.15.71:0/3208673296 shutdown_connections 2024-03-29T11:02:33.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:33.107+0000 7fb23caa8640 1 -- 172.21.15.71:0/3208673296 >> 172.21.15.71:0/3208673296 conn(0x7fb238066a00 msgr2=0x7fb2380b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:33.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:33.107+0000 7fb23caa8640 1 -- 172.21.15.71:0/3208673296 shutdown_connections 2024-03-29T11:02:33.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:33.107+0000 7fb23caa8640 1 -- 172.21.15.71:0/3208673296 wait complete. 2024-03-29T11:02:33.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:33.108+0000 7fb23caa8640 1 librados: shutdown 2024-03-29T11:02:33.104 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:39.911 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:40.294 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.313+0000 7f25fe2db640 10 monclient: get_monmap_and_config 2024-03-29T11:02:40.294 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.313+0000 7f25fe2db640 10 monclient: build_initial_monmap 2024-03-29T11:02:40.294 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.313+0000 7f25fe2db640 10 monclient: monmap: 2024-03-29T11:02:40.295 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:02:40.295 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:40.295 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:02:40.315670+0000 2024-03-29T11:02:40.295 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:02:40.315670+0000 2024-03-29T11:02:40.295 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:40.295 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:40.295 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:02:40.295 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:02:40.295 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:02:40.295 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:40.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.316+0000 7f25fe2db640 1 Processor -- start 2024-03-29T11:02:40.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.316+0000 7f25fe2db640 1 -- start start 2024-03-29T11:02:40.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.316+0000 7f25fe2db640 10 monclient: init 2024-03-29T11:02:40.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.316+0000 7f25fe2db640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:40.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.317+0000 7f25fe2db640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:02:40.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.317+0000 7f25fe2db640 10 monclient(hunting): picked mon.noname-c con 0x7f25f80b0860 addr v1:172.21.15.137:6789/0 2024-03-29T11:02:40.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.317+0000 7f25fe2db640 10 monclient(hunting): picked mon.noname-b con 0x7f25f813ef30 addr v1:172.21.15.71:6790/0 2024-03-29T11:02:40.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.317+0000 7f25fe2db640 10 monclient(hunting): picked mon.noname-a con 0x7f25f80af8e0 addr v1:172.21.15.71:6789/0 2024-03-29T11:02:40.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.317+0000 7f25fe2db640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f80b0c40 con 0x7f25f80af8e0 2024-03-29T11:02:40.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.317+0000 7f25fe2db640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f80aed10 con 0x7f25f80b0860 2024-03-29T11:02:40.300 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.317+0000 7f25fe2db640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f80afd50 con 0x7f25f813ef30 2024-03-29T11:02:40.300 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.318+0000 7f25fe2db640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:40.300 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.318+0000 7f25fe2db640 10 monclient(hunting): authenticate will time out at 916.578055s 2024-03-29T11:02:40.300 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.320+0000 7f25f7fff640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f25f80b0860 0x7f25f813e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:42793/0 (socket says 172.21.15.71:42793) 2024-03-29T11:02:40.301 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.320+0000 7f25f7fff640 1 -- 172.21.15.71:0/1402163968 learned_addr learned my addr 172.21.15.71:0/1402163968 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:02:40.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.322+0000 7f25f6ffd640 1 -- 172.21.15.71:0/1402163968 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415011060 0 0) 0x7f25f80aed10 con 0x7f25f80b0860 2024-03-29T11:02:40.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.322+0000 7f25f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:40.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.322+0000 7f25f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:40.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.322+0000 7f25f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:40.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.322+0000 7f25f6ffd640 10 monclient(hunting): my global_id is 4285 2024-03-29T11:02:40.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.322+0000 7f25f6ffd640 1 -- 172.21.15.71:0/1402163968 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25e0003520 con 0x7f25f80b0860 2024-03-29T11:02:40.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.323+0000 7f25f6ffd640 1 -- 172.21.15.71:0/1402163968 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4041847731 0 0) 0x7f25f80b0c40 con 0x7f25f80af8e0 2024-03-29T11:02:40.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:40.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:40.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:40.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 10 monclient(hunting): my global_id is 4317 2024-03-29T11:02:40.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 1 -- 172.21.15.71:0/1402163968 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25e0003840 con 0x7f25f80af8e0 2024-03-29T11:02:40.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 1 -- 172.21.15.71:0/1402163968 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1805332377 0 0) 0x7f25f80afd50 con 0x7f25f813ef30 2024-03-29T11:02:40.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:40.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:40.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:40.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 10 monclient(hunting): my global_id is 4304 2024-03-29T11:02:40.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 1 -- 172.21.15.71:0/1402163968 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25f80b0c40 con 0x7f25f813ef30 2024-03-29T11:02:40.305 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 1 -- 172.21.15.71:0/1402163968 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 174284667 0 0) 0x7f25e0003520 con 0x7f25f80b0860 2024-03-29T11:02:40.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 1 -- 172.21.15.71:0/1402163968 >> v1:172.21.15.71:6790/0 conn(0x7f25f813ef30 legacy=0x7f25f80af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:40.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 1 -- 172.21.15.71:0/1402163968 >> v1:172.21.15.71:6789/0 conn(0x7f25f80af8e0 legacy=0x7f25f8149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:40.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:02:40.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 1 monclient: found mon.noname-c 2024-03-29T11:02:40.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:40.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.137:6789/0 2024-03-29T11:02:40.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 1 -- 172.21.15.71:0/1402163968 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f80ab1b0 con 0x7f25f80b0860 2024-03-29T11:02:40.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:02:40.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:02:40.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:40.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 1 -- 172.21.15.71:0/1402163968 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f25e80015f0 con 0x7f25f80b0860 2024-03-29T11:02:40.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:40.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:40.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 10 monclient: dump: 2024-03-29T11:02:40.307 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:40.307 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:40.307 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:40.307 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:40.307 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:40.307 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:40.307 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:40.307 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:40.307 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:40.307 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:40.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25fe2db640 5 monclient: authenticate success, global_id 4285 2024-03-29T11:02:40.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25fe2db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:40.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 1 -- 172.21.15.71:0/1402163968 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25e8001750 con 0x7f25f80b0860 2024-03-29T11:02:40.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:40.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 1 -- 172.21.15.71:0/1402163968 <== mon.1 v1:172.21.15.137:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f25e8001ba0 con 0x7f25f80b0860 2024-03-29T11:02:40.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:40.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:02:40.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25f6ffd640 10 monclient: dump: 2024-03-29T11:02:40.308 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:40.308 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:40.308 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:40.308 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:40.308 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:40.308 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:40.308 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:40.308 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:40.308 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:40.308 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:40.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25fe2db640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:40.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.324+0000 7f25fe2db640 4 set_mon_vals no callback set 2024-03-29T11:02:40.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.325+0000 7f25fe2db640 10 monclient: shutdown 2024-03-29T11:02:40.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.325+0000 7f25fe2db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:40.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.325+0000 7f25fe2db640 1 -- 172.21.15.71:0/1402163968 >> v1:172.21.15.137:6789/0 conn(0x7f25f80b0860 legacy=0x7f25f813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:40.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.325+0000 7f25fe2db640 1 -- 172.21.15.71:0/1402163968 shutdown_connections 2024-03-29T11:02:40.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.325+0000 7f25fe2db640 1 -- 172.21.15.71:0/1402163968 >> 172.21.15.71:0/1402163968 conn(0x7f25f806b660 msgr2=0x7f25f806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:40.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.325+0000 7f25fe2db640 1 -- 172.21.15.71:0/1402163968 shutdown_connections 2024-03-29T11:02:40.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.325+0000 7f25fe2db640 1 -- 172.21.15.71:0/1402163968 wait complete. 2024-03-29T11:02:40.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.327+0000 7f25fe2db640 10 monclient: build_initial_monmap 2024-03-29T11:02:40.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.327+0000 7f25fe2db640 10 monclient: monmap: 2024-03-29T11:02:40.311 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:02:40.311 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:40.311 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:02:40.311 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:02:40.311 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:40.311 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:40.311 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:02:40.311 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:02:40.311 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:02:40.312 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:40.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.327+0000 7f25fe2db640 1 librados: starting msgr at 2024-03-29T11:02:40.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.327+0000 7f25fe2db640 1 librados: starting objecter 2024-03-29T11:02:40.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.327+0000 7f25fe2db640 1 Processor -- start 2024-03-29T11:02:40.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.328+0000 7f25fe2db640 1 -- start start 2024-03-29T11:02:40.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.328+0000 7f25fe2db640 1 librados: setting wanted keys 2024-03-29T11:02:40.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.328+0000 7f25fe2db640 1 librados: calling monclient init 2024-03-29T11:02:40.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.328+0000 7f25fe2db640 10 monclient: init 2024-03-29T11:02:40.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.329+0000 7f25fe2db640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:40.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.329+0000 7f25fe2db640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:02:40.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.329+0000 7f25fe2db640 10 monclient(hunting): picked mon.noname-c con 0x7f25f80af8e0 addr v1:172.21.15.71:6790/0 2024-03-29T11:02:40.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.330+0000 7f25fe2db640 10 monclient(hunting): picked mon.noname-b con 0x7f25f80b0860 addr v1:172.21.15.137:6789/0 2024-03-29T11:02:40.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.332+0000 7f25fe2db640 10 monclient(hunting): picked mon.noname-a con 0x7f25f813ef30 addr v1:172.21.15.71:6789/0 2024-03-29T11:02:40.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.332+0000 7f25fe2db640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f813b280 con 0x7f25f813ef30 2024-03-29T11:02:40.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.332+0000 7f25fe2db640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f813a2b0 con 0x7f25f80b0860 2024-03-29T11:02:40.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.332+0000 7f25fe2db640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f8139f40 con 0x7f25f80af8e0 2024-03-29T11:02:40.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.332+0000 7f25fe2db640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:40.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.332+0000 7f25fe2db640 10 monclient(hunting): authenticate will time out at 916.591628s 2024-03-29T11:02:40.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.339+0000 7f25fc878640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f25f813ef30 0x7f25f81d6d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:41767/0 (socket says 172.21.15.71:41767) 2024-03-29T11:02:40.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.339+0000 7f25fc878640 1 -- 172.21.15.71:0/499143394 learned_addr learned my addr 172.21.15.71:0/499143394 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:02:40.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.340+0000 7f25f4ff9640 10 client.?.objecter ms_handle_connect 0x7f25f80b0860 2024-03-29T11:02:40.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.340+0000 7f25f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:40.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.340+0000 7f25f4ff9640 1 -- 172.21.15.71:0/499143394 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285426691 0 0) 0x7f25f813a2b0 con 0x7f25f80b0860 2024-03-29T11:02:40.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.340+0000 7f25f4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:02:40.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.340+0000 7f25f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:40.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.340+0000 7f25f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:40.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.340+0000 7f25f4ff9640 10 monclient(hunting): my global_id is 4288 2024-03-29T11:02:40.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.340+0000 7f25f4ff9640 1 -- 172.21.15.71:0/499143394 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25e4004eb0 con 0x7f25f80b0860 2024-03-29T11:02:40.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.340+0000 7f25f4ff9640 10 client.?.objecter ms_handle_connect 0x7f25f813ef30 2024-03-29T11:02:40.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.340+0000 7f25f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:40.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.340+0000 7f25f4ff9640 10 client.?.objecter ms_handle_connect 0x7f25f80af8e0 2024-03-29T11:02:40.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.340+0000 7f25f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:40.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.340+0000 7f25f4ff9640 1 -- 172.21.15.71:0/499143394 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2016353683 0 0) 0x7f25f813b280 con 0x7f25f813ef30 2024-03-29T11:02:40.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.340+0000 7f25f4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:02:40.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.340+0000 7f25f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:40.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.340+0000 7f25f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:40.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.340+0000 7f25f4ff9640 10 monclient(hunting): my global_id is 4320 2024-03-29T11:02:40.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.340+0000 7f25f4ff9640 1 -- 172.21.15.71:0/499143394 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25f813a2b0 con 0x7f25f813ef30 2024-03-29T11:02:40.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.341+0000 7f25f4ff9640 1 -- 172.21.15.71:0/499143394 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2845319192 0 0) 0x7f25e4004eb0 con 0x7f25f80b0860 2024-03-29T11:02:40.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.341+0000 7f25f4ff9640 1 -- 172.21.15.71:0/499143394 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25f813b280 con 0x7f25f80b0860 2024-03-29T11:02:40.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.341+0000 7f25f4ff9640 1 -- 172.21.15.71:0/499143394 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f25f0001b00 con 0x7f25f80b0860 2024-03-29T11:02:40.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.341+0000 7f25f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:40.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.341+0000 7f25f4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:40.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.341+0000 7f25f4ff9640 10 monclient(hunting): dump: 2024-03-29T11:02:40.326 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:40.326 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:40.327 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:40.327 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:40.327 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:40.327 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:40.327 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:40.327 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:40.327 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:40.327 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:40.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.341+0000 7f25f4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:40.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.341+0000 7f25f4ff9640 1 -- 172.21.15.71:0/499143394 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3897672730 0 0) 0x7f25f813a2b0 con 0x7f25f813ef30 2024-03-29T11:02:40.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.341+0000 7f25f4ff9640 1 -- 172.21.15.71:0/499143394 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25e4004eb0 con 0x7f25f813ef30 2024-03-29T11:02:40.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.341+0000 7f25f4ff9640 1 -- 172.21.15.71:0/499143394 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f25ec001970 con 0x7f25f813ef30 2024-03-29T11:02:40.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.341+0000 7f25f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:40.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.342+0000 7f25f4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:40.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.342+0000 7f25f4ff9640 10 monclient(hunting): dump: 2024-03-29T11:02:40.328 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:40.328 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:40.328 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:40.328 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:40.328 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:40.328 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:40.328 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:40.328 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:40.328 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:40.328 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:40.328 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.342+0000 7f25f4ff9640 1 -- 172.21.15.71:0/499143394 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2857735383 0 0) 0x7f25f813b280 con 0x7f25f80b0860 2024-03-29T11:02:40.328 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.342+0000 7f25f4ff9640 1 -- 172.21.15.71:0/499143394 >> v1:172.21.15.71:6790/0 conn(0x7f25f80af8e0 legacy=0x7f25f8139830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:40.328 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.342+0000 7f25f4ff9640 1 -- 172.21.15.71:0/499143394 >> v1:172.21.15.71:6789/0 conn(0x7f25f813ef30 legacy=0x7f25f81d6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:40.328 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.342+0000 7f25f4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:02:40.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.342+0000 7f25f4ff9640 1 monclient: found mon.b 2024-03-29T11:02:40.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.342+0000 7f25f4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:40.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.342+0000 7f25f4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:02:40.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.342+0000 7f25f4ff9640 1 -- 172.21.15.71:0/499143394 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f813b8f0 con 0x7f25f80b0860 2024-03-29T11:02:40.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.342+0000 7f25f4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:02:40.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.342+0000 7f25f4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:02:40.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.342+0000 7f25f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:40.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.342+0000 7f25fe2db640 5 monclient: authenticate success, global_id 4288 2024-03-29T11:02:40.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.342+0000 7f25fe2db640 10 monclient: _renew_subs 2024-03-29T11:02:40.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.342+0000 7f25fe2db640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:02:40.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.342+0000 7f25fe2db640 1 -- 172.21.15.71:0/499143394 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25f813bba0 con 0x7f25f80b0860 2024-03-29T11:02:40.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.343+0000 7f25fe2db640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:40.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.343+0000 7f25fe2db640 10 monclient: _renew_subs 2024-03-29T11:02:40.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.343+0000 7f25fe2db640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:02:40.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.343+0000 7f25fe2db640 1 -- 172.21.15.71:0/499143394 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f81deb80 con 0x7f25f80b0860 2024-03-29T11:02:40.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.343+0000 7f25fe2db640 1 librados: init done 2024-03-29T11:02:40.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.343+0000 7f25f4ff9640 1 -- 172.21.15.71:0/499143394 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25f0002010 con 0x7f25f80b0860 2024-03-29T11:02:40.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.343+0000 7f25f4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:40.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.343+0000 7f25f4ff9640 1 -- 172.21.15.71:0/499143394 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f25f0003b10 con 0x7f25f80b0860 2024-03-29T11:02:40.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.343+0000 7f25f5ffb640 4 set_mon_vals no callback set 2024-03-29T11:02:40.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.344+0000 7f25f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:40.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.345+0000 7f25f4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:02:40.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.345+0000 7f25f4ff9640 10 monclient: dump: 2024-03-29T11:02:40.331 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:40.331 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:40.331 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:40.331 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:40.331 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:40.331 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:40.331 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:40.331 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:40.331 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:40.331 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:40.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.347+0000 7f25fe2db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:40.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.347+0000 7f25fe2db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:40.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.347+0000 7f25fe2db640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:02:40.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.347+0000 7f25fe2db640 1 -- 172.21.15.71:0/499143394 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25d8005080 con 0x7f25f80b0860 2024-03-29T11:02:40.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.349+0000 7f25f4ff9640 1 -- 172.21.15.71:0/499143394 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f25f0019ae0 con 0x7f25f80b0860 2024-03-29T11:02:40.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.349+0000 7f25f4ff9640 1 -- 172.21.15.71:0/499143394 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f25f0083980 con 0x7f25f80b0860 2024-03-29T11:02:40.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.349+0000 7f25f4ff9640 10 client.4288.objecter ms_dispatch 0x7f25f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:02:40.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.349+0000 7f25f4ff9640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:02:40.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.349+0000 7f25f4ff9640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:02:40.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.349+0000 7f25f4ff9640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-29T11:02:40.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.349+0000 7f25f7fff640 10 monclient: get_auth_request con 0x7f25e406dec0 auth_method 0 2024-03-29T11:02:40.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.352+0000 7f25f4ff9640 10 client.4288.objecter ms_handle_connect 0x7f25e406dec0 2024-03-29T11:02:40.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.357+0000 7f25f4ff9640 1 -- 172.21.15.71:0/499143394 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f25f004f260 con 0x7f25f80b0860 2024-03-29T11:02:40.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.357+0000 7f25f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:40.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.357+0000 7f25f4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:40.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.678+0000 7f25fe2db640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:40.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.678+0000 7f25fe2db640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:40.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.678+0000 7f25fe2db640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:02:40.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.678+0000 7f25fe2db640 1 -- 172.21.15.71:0/499143394 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25d8005330 con 0x7f25f80b0860 2024-03-29T11:02:40.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.679+0000 7f25f4ff9640 1 -- 172.21.15.71:0/499143394 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f25f0051190 con 0x7f25f80b0860 2024-03-29T11:02:40.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.679+0000 7f25f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:40.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.679+0000 7f25f4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:40.664 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:02:40.664 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:40.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.687+0000 7f25fe2db640 10 librados: watch_flush enter 2024-03-29T11:02:40.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.687+0000 7f25fe2db640 10 librados: watch_flush exit 2024-03-29T11:02:40.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.687+0000 7f25fe2db640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-29T11:02:40.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.687+0000 7f25fe2db640 10 client.4288.objecter successfully canceled tick 2024-03-29T11:02:40.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.687+0000 7f25fe2db640 1 -- 172.21.15.71:0/499143394 >> v1:172.21.15.137:6816/973332665 conn(0x7f25e406dec0 legacy=0x7f25e4070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:40.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.687+0000 7f25fe2db640 10 monclient: shutdown 2024-03-29T11:02:40.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.687+0000 7f25fe2db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:40.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.687+0000 7f25fe2db640 1 -- 172.21.15.71:0/499143394 >> v1:172.21.15.137:6789/0 conn(0x7f25f80b0860 legacy=0x7f25f81d6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:40.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.688+0000 7f25fe2db640 1 -- 172.21.15.71:0/499143394 shutdown_connections 2024-03-29T11:02:40.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.688+0000 7f25fe2db640 1 -- 172.21.15.71:0/499143394 >> 172.21.15.71:0/499143394 conn(0x7f25f806b660 msgr2=0x7f25f81c8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:40.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.688+0000 7f25fe2db640 1 -- 172.21.15.71:0/499143394 shutdown_connections 2024-03-29T11:02:40.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.689+0000 7f25fe2db640 1 -- 172.21.15.71:0/499143394 wait complete. 2024-03-29T11:02:40.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:40.689+0000 7f25fe2db640 1 librados: shutdown 2024-03-29T11:02:40.686 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:47.493 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:47.658 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.678+0000 7febfb59e640 10 monclient: get_monmap_and_config 2024-03-29T11:02:47.658 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.678+0000 7febfb59e640 10 monclient: build_initial_monmap 2024-03-29T11:02:47.658 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.678+0000 7febfb59e640 10 monclient: monmap: 2024-03-29T11:02:47.658 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:02:47.658 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:47.658 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:02:47.679455+0000 2024-03-29T11:02:47.658 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:02:47.679455+0000 2024-03-29T11:02:47.658 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:47.658 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:47.658 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:02:47.659 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:02:47.659 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:02:47.659 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:47.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.679+0000 7febfb59e640 1 Processor -- start 2024-03-29T11:02:47.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.679+0000 7febfb59e640 1 -- start start 2024-03-29T11:02:47.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.679+0000 7febfb59e640 10 monclient: init 2024-03-29T11:02:47.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.680+0000 7febfb59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:47.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.680+0000 7febfb59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:02:47.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.680+0000 7febfb59e640 10 monclient(hunting): picked mon.noname-b con 0x7febf4005ef0 addr v1:172.21.15.71:6790/0 2024-03-29T11:02:47.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.681+0000 7febfb59e640 10 monclient(hunting): picked mon.noname-c con 0x7febf40cf3c0 addr v1:172.21.15.137:6789/0 2024-03-29T11:02:47.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.681+0000 7febfb59e640 10 monclient(hunting): picked mon.noname-a con 0x7febf4033640 addr v1:172.21.15.71:6789/0 2024-03-29T11:02:47.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.681+0000 7febfb59e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf4032c60 con 0x7febf4033640 2024-03-29T11:02:47.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.681+0000 7febfb59e640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf4032f40 con 0x7febf40cf3c0 2024-03-29T11:02:47.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.681+0000 7febfb59e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf402cba0 con 0x7febf4005ef0 2024-03-29T11:02:47.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.681+0000 7febfb59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:47.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.681+0000 7febfb59e640 10 monclient(hunting): authenticate will time out at 923.939176s 2024-03-29T11:02:47.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.681+0000 7febfa59c640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7febf4005ef0 0x7febf4031500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:40096/0 (socket says 172.21.15.71:40096) 2024-03-29T11:02:47.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.681+0000 7febfa59c640 1 -- 172.21.15.71:0/2853050147 learned_addr learned my addr 172.21.15.71:0/2853050147 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:02:47.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.682+0000 7febf959a640 1 -- 172.21.15.71:0/2853050147 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448126921 0 0) 0x7febf402cba0 con 0x7febf4005ef0 2024-03-29T11:02:47.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.682+0000 7febf959a640 10 monclient(hunting): _negotiate 2024-03-29T11:02:47.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.682+0000 7febf959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:47.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.682+0000 7febf959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:47.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.682+0000 7febf959a640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:02:47.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.682+0000 7febf959a640 1 -- 172.21.15.71:0/2853050147 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febe4003520 con 0x7febf4005ef0 2024-03-29T11:02:47.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.682+0000 7febf959a640 1 -- 172.21.15.71:0/2853050147 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789080987 0 0) 0x7febf4032f40 con 0x7febf40cf3c0 2024-03-29T11:02:47.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.682+0000 7febf959a640 10 monclient(hunting): _negotiate 2024-03-29T11:02:47.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.682+0000 7febf959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:47.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.682+0000 7febf959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:47.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.682+0000 7febf959a640 10 monclient(hunting): my global_id is 4291 2024-03-29T11:02:47.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.682+0000 7febf959a640 1 -- 172.21.15.71:0/2853050147 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febf402cba0 con 0x7febf40cf3c0 2024-03-29T11:02:47.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.682+0000 7febf959a640 1 -- 172.21.15.71:0/2853050147 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837993070 0 0) 0x7febf4032c60 con 0x7febf4033640 2024-03-29T11:02:47.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.682+0000 7febf959a640 10 monclient(hunting): _negotiate 2024-03-29T11:02:47.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.682+0000 7febf959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:47.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.682+0000 7febf959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:47.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.682+0000 7febf959a640 10 monclient(hunting): my global_id is 4323 2024-03-29T11:02:47.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.682+0000 7febf959a640 1 -- 172.21.15.71:0/2853050147 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febf4032f40 con 0x7febf4033640 2024-03-29T11:02:47.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.683+0000 7febf959a640 1 -- 172.21.15.71:0/2853050147 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2880370875 0 0) 0x7febf402cba0 con 0x7febf40cf3c0 2024-03-29T11:02:47.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.683+0000 7febf959a640 1 -- 172.21.15.71:0/2853050147 >> v1:172.21.15.71:6790/0 conn(0x7febf4005ef0 legacy=0x7febf4031500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:47.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.683+0000 7febf959a640 1 -- 172.21.15.71:0/2853050147 >> v1:172.21.15.71:6789/0 conn(0x7febf4033640 legacy=0x7febf40d7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:47.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.683+0000 7febf959a640 10 monclient: _finish_hunting 0 2024-03-29T11:02:47.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.683+0000 7febf959a640 1 monclient: found mon.noname-c 2024-03-29T11:02:47.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.683+0000 7febf959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:47.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.683+0000 7febf959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.137:6789/0 2024-03-29T11:02:47.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.683+0000 7febf959a640 1 -- 172.21.15.71:0/2853050147 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf4032080 con 0x7febf40cf3c0 2024-03-29T11:02:47.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.683+0000 7febf959a640 10 monclient: _finish_auth 0 2024-03-29T11:02:47.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.683+0000 7febf959a640 10 monclient: _check_auth_tickets 2024-03-29T11:02:47.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.683+0000 7febf959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:47.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.683+0000 7febfb59e640 5 monclient: authenticate success, global_id 4291 2024-03-29T11:02:47.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.683+0000 7febfb59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:47.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.683+0000 7febf959a640 1 -- 172.21.15.71:0/2853050147 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7febf0001670 con 0x7febf40cf3c0 2024-03-29T11:02:47.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.683+0000 7febf959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:47.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.683+0000 7febf959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:47.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.683+0000 7febf959a640 10 monclient: dump: 2024-03-29T11:02:47.666 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:47.666 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:47.666 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:47.666 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:47.666 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:47.666 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:47.666 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:47.666 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:47.666 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:47.666 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:47.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.683+0000 7febfb59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:47.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.683+0000 7febf959a640 1 -- 172.21.15.71:0/2853050147 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febf00017d0 con 0x7febf40cf3c0 2024-03-29T11:02:47.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.683+0000 7febf959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:47.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.683+0000 7febfb59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:47.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.683+0000 7febfb59e640 4 set_mon_vals no callback set 2024-03-29T11:02:47.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.684+0000 7febf959a640 1 -- 172.21.15.71:0/2853050147 <== mon.1 v1:172.21.15.137:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7febf0001ba0 con 0x7febf40cf3c0 2024-03-29T11:02:47.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.684+0000 7febf959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:47.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.684+0000 7febf959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:02:47.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.684+0000 7febf959a640 10 monclient: dump: 2024-03-29T11:02:47.667 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:47.667 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:47.667 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:47.667 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:47.667 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:47.667 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:47.667 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:47.667 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:47.667 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:47.668 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:47.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.685+0000 7febfb59e640 10 monclient: shutdown 2024-03-29T11:02:47.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.685+0000 7febfb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:47.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.685+0000 7febfb59e640 1 -- 172.21.15.71:0/2853050147 >> v1:172.21.15.137:6789/0 conn(0x7febf40cf3c0 legacy=0x7febf40cc280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:47.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.685+0000 7febfb59e640 1 -- 172.21.15.71:0/2853050147 shutdown_connections 2024-03-29T11:02:47.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.685+0000 7febfb59e640 1 -- 172.21.15.71:0/2853050147 >> 172.21.15.71:0/2853050147 conn(0x7febf4006510 msgr2=0x7febf40068f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:47.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.685+0000 7febfb59e640 1 -- 172.21.15.71:0/2853050147 shutdown_connections 2024-03-29T11:02:47.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.685+0000 7febfb59e640 1 -- 172.21.15.71:0/2853050147 wait complete. 2024-03-29T11:02:47.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.686+0000 7febfb59e640 10 monclient: build_initial_monmap 2024-03-29T11:02:47.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.686+0000 7febfb59e640 10 monclient: monmap: 2024-03-29T11:02:47.673 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:02:47.673 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:47.673 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:02:47.673 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:02:47.673 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:47.673 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:47.673 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:02:47.673 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:02:47.673 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:02:47.674 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:47.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.686+0000 7febfb59e640 1 librados: starting msgr at 2024-03-29T11:02:47.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.686+0000 7febfb59e640 1 librados: starting objecter 2024-03-29T11:02:47.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.687+0000 7febfb59e640 1 Processor -- start 2024-03-29T11:02:47.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.687+0000 7febfb59e640 1 -- start start 2024-03-29T11:02:47.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.687+0000 7febfb59e640 1 librados: setting wanted keys 2024-03-29T11:02:47.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.687+0000 7febfb59e640 1 librados: calling monclient init 2024-03-29T11:02:47.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.687+0000 7febfb59e640 10 monclient: init 2024-03-29T11:02:47.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.687+0000 7febfb59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:47.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.687+0000 7febfb59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:02:47.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.687+0000 7febfb59e640 10 monclient(hunting): picked mon.noname-c con 0x7febf4005ef0 addr v1:172.21.15.71:6790/0 2024-03-29T11:02:47.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.687+0000 7febfb59e640 10 monclient(hunting): picked mon.noname-b con 0x7febf4033640 addr v1:172.21.15.137:6789/0 2024-03-29T11:02:47.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.687+0000 7febfb59e640 10 monclient(hunting): picked mon.noname-a con 0x7febf40cf3c0 addr v1:172.21.15.71:6789/0 2024-03-29T11:02:47.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.687+0000 7febfb59e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf402bb60 con 0x7febf40cf3c0 2024-03-29T11:02:47.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.687+0000 7febfb59e640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf402aeb0 con 0x7febf4033640 2024-03-29T11:02:47.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.687+0000 7febfb59e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf402b040 con 0x7febf4005ef0 2024-03-29T11:02:47.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.687+0000 7febfb59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:47.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.687+0000 7febfb59e640 10 monclient(hunting): authenticate will time out at 923.946011s 2024-03-29T11:02:47.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.688+0000 7febfad9d640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7febf40cf3c0 0x7febf41642a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:38010/0 (socket says 172.21.15.71:38010) 2024-03-29T11:02:47.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.688+0000 7febfad9d640 1 -- 172.21.15.71:0/2782507331 learned_addr learned my addr 172.21.15.71:0/2782507331 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:02:47.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.688+0000 7febeaffd640 10 client.?.objecter ms_handle_connect 0x7febf40cf3c0 2024-03-29T11:02:47.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.688+0000 7febeaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:47.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.688+0000 7febeaffd640 10 client.?.objecter ms_handle_connect 0x7febf4033640 2024-03-29T11:02:47.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.688+0000 7febeaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:47.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.688+0000 7febeaffd640 1 -- 172.21.15.71:0/2782507331 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2229520411 0 0) 0x7febf402aeb0 con 0x7febf4033640 2024-03-29T11:02:47.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.688+0000 7febeaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:47.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.688+0000 7febeaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:47.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.688+0000 7febeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:47.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.688+0000 7febeaffd640 10 monclient(hunting): my global_id is 4294 2024-03-29T11:02:47.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.688+0000 7febeaffd640 1 -- 172.21.15.71:0/2782507331 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febdc004eb0 con 0x7febf4033640 2024-03-29T11:02:47.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.688+0000 7febeaffd640 10 client.?.objecter ms_handle_connect 0x7febf4005ef0 2024-03-29T11:02:47.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.688+0000 7febeaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:47.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.689+0000 7febeaffd640 1 -- 172.21.15.71:0/2782507331 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1978656550 0 0) 0x7febf402bb60 con 0x7febf40cf3c0 2024-03-29T11:02:47.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.689+0000 7febeaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:47.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.689+0000 7febeaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:47.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.689+0000 7febeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:47.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.689+0000 7febeaffd640 10 monclient(hunting): my global_id is 4326 2024-03-29T11:02:47.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.689+0000 7febeaffd640 1 -- 172.21.15.71:0/2782507331 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febf402aeb0 con 0x7febf40cf3c0 2024-03-29T11:02:47.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.689+0000 7febeaffd640 1 -- 172.21.15.71:0/2782507331 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3089500945 0 0) 0x7febdc004eb0 con 0x7febf4033640 2024-03-29T11:02:47.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.689+0000 7febeaffd640 1 -- 172.21.15.71:0/2782507331 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febf402bb60 con 0x7febf4033640 2024-03-29T11:02:47.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.689+0000 7febeaffd640 1 -- 172.21.15.71:0/2782507331 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7febf0001ba0 con 0x7febf4033640 2024-03-29T11:02:47.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.689+0000 7febeaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:47.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.689+0000 7febeaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:47.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.689+0000 7febeaffd640 10 monclient(hunting): dump: 2024-03-29T11:02:47.676 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:47.676 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:47.677 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:47.677 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:47.677 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:47.677 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:47.677 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:47.677 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:47.677 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:47.677 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:47.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.689+0000 7febeaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:47.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.689+0000 7febeaffd640 1 -- 172.21.15.71:0/2782507331 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704884739 0 0) 0x7febf402b040 con 0x7febf4005ef0 2024-03-29T11:02:47.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.689+0000 7febeaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:47.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.689+0000 7febeaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:47.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.689+0000 7febeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:47.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.689+0000 7febeaffd640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:02:47.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.689+0000 7febeaffd640 1 -- 172.21.15.71:0/2782507331 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febdc005780 con 0x7febf4005ef0 2024-03-29T11:02:47.678 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febeaffd640 1 -- 172.21.15.71:0/2782507331 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2008454044 0 0) 0x7febf402aeb0 con 0x7febf40cf3c0 2024-03-29T11:02:47.678 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febeaffd640 1 -- 172.21.15.71:0/2782507331 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febf402b040 con 0x7febf40cf3c0 2024-03-29T11:02:47.678 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febeaffd640 1 -- 172.21.15.71:0/2782507331 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7febfc04fdd0 con 0x7febf40cf3c0 2024-03-29T11:02:47.678 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febeaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:47.678 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febeaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:47.678 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febeaffd640 10 monclient(hunting): dump: 2024-03-29T11:02:47.678 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:47.678 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:47.678 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:47.678 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:47.678 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:47.678 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:47.678 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:47.679 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:47.679 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:47.679 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:47.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febeaffd640 1 -- 172.21.15.71:0/2782507331 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1512701150 0 0) 0x7febf402bb60 con 0x7febf4033640 2024-03-29T11:02:47.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febeaffd640 1 -- 172.21.15.71:0/2782507331 >> v1:172.21.15.71:6790/0 conn(0x7febf4005ef0 legacy=0x7febf402b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:47.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febeaffd640 1 -- 172.21.15.71:0/2782507331 >> v1:172.21.15.71:6789/0 conn(0x7febf40cf3c0 legacy=0x7febf41642a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:47.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febeaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:02:47.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febeaffd640 1 monclient: found mon.b 2024-03-29T11:02:47.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febeaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:47.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febeaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:02:47.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febeaffd640 1 -- 172.21.15.71:0/2782507331 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf40c7a20 con 0x7febf4033640 2024-03-29T11:02:47.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febeaffd640 10 monclient: _finish_auth 0 2024-03-29T11:02:47.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febeaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:02:47.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:47.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febfb59e640 5 monclient: authenticate success, global_id 4294 2024-03-29T11:02:47.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febfb59e640 10 monclient: _renew_subs 2024-03-29T11:02:47.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febfb59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:02:47.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febfb59e640 1 -- 172.21.15.71:0/2782507331 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf40c7cd0 con 0x7febf4033640 2024-03-29T11:02:47.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febfb59e640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:47.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febfb59e640 10 monclient: _renew_subs 2024-03-29T11:02:47.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febfb59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:02:47.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febfb59e640 1 -- 172.21.15.71:0/2782507331 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febf402f140 con 0x7febf4033640 2024-03-29T11:02:47.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febfb59e640 1 librados: init done 2024-03-29T11:02:47.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febeaffd640 1 -- 172.21.15.71:0/2782507331 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febf00038a0 con 0x7febf4033640 2024-03-29T11:02:47.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febeaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:47.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febf8d99640 4 set_mon_vals no callback set 2024-03-29T11:02:47.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febeaffd640 1 -- 172.21.15.71:0/2782507331 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7febf0003bb0 con 0x7febf4033640 2024-03-29T11:02:47.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.690+0000 7febeaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:47.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.691+0000 7febeaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:02:47.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.691+0000 7febeaffd640 10 monclient: dump: 2024-03-29T11:02:47.681 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:47.681 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:47.681 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:47.681 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:47.681 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:47.681 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:47.681 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:47.681 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:47.681 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:47.681 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:47.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.692+0000 7febfb59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:47.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.692+0000 7febfb59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:47.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.692+0000 7febfb59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:02:47.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.692+0000 7febfb59e640 1 -- 172.21.15.71:0/2782507331 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe0005080 con 0x7febf4033640 2024-03-29T11:02:47.682 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.692+0000 7febeaffd640 1 -- 172.21.15.71:0/2782507331 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7febf0019b80 con 0x7febf4033640 2024-03-29T11:02:47.682 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.693+0000 7febeaffd640 1 -- 172.21.15.71:0/2782507331 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7febf0083910 con 0x7febf4033640 2024-03-29T11:02:47.682 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.693+0000 7febeaffd640 10 client.4294.objecter ms_dispatch 0x7febfc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:02:47.682 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.693+0000 7febeaffd640 3 client.4294.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:02:47.682 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.693+0000 7febeaffd640 3 client.4294.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:02:47.682 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.693+0000 7febeaffd640 20 client.4294.objecter dump_active .. 0 homeless 2024-03-29T11:02:47.682 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.693+0000 7febfa59c640 10 monclient: get_auth_request con 0x7febdc06de40 auth_method 0 2024-03-29T11:02:47.682 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.693+0000 7febeaffd640 10 client.4294.objecter ms_handle_connect 0x7febdc06de40 2024-03-29T11:02:47.682 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.700+0000 7febeaffd640 1 -- 172.21.15.71:0/2782507331 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7febf004f1f0 con 0x7febf4033640 2024-03-29T11:02:47.682 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.700+0000 7febeaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:47.683 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:47.700+0000 7febeaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:48.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:48.096+0000 7febfb59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:48.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:48.097+0000 7febfb59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:48.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:48.097+0000 7febfb59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:02:48.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:48.097+0000 7febfb59e640 1 -- 172.21.15.71:0/2782507331 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febe0005330 con 0x7febf4033640 2024-03-29T11:02:48.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:48.098+0000 7febeaffd640 1 -- 172.21.15.71:0/2782507331 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7febf0051120 con 0x7febf4033640 2024-03-29T11:02:48.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:48.098+0000 7febeaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:48.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:48.098+0000 7febeaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:48.078 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:02:48.078 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:48.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:48.102+0000 7febfb59e640 10 librados: watch_flush enter 2024-03-29T11:02:48.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:48.102+0000 7febfb59e640 10 librados: watch_flush exit 2024-03-29T11:02:48.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:48.102+0000 7febfb59e640 20 client.4294.objecter shutdown clearing up homeless session... 2024-03-29T11:02:48.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:48.102+0000 7febfb59e640 10 client.4294.objecter successfully canceled tick 2024-03-29T11:02:48.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:48.102+0000 7febfb59e640 1 -- 172.21.15.71:0/2782507331 >> v1:172.21.15.137:6816/973332665 conn(0x7febdc06de40 legacy=0x7febdc070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:48.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:48.102+0000 7febfb59e640 10 monclient: shutdown 2024-03-29T11:02:48.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:48.102+0000 7febfb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:48.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:48.102+0000 7febfb59e640 1 -- 172.21.15.71:0/2782507331 >> v1:172.21.15.137:6789/0 conn(0x7febf4033640 legacy=0x7febf40c7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:48.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:48.103+0000 7febfb59e640 1 -- 172.21.15.71:0/2782507331 shutdown_connections 2024-03-29T11:02:48.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:48.103+0000 7febfb59e640 1 -- 172.21.15.71:0/2782507331 >> 172.21.15.71:0/2782507331 conn(0x7febf4006510 msgr2=0x7febf40cc050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:48.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:48.103+0000 7febfb59e640 1 -- 172.21.15.71:0/2782507331 shutdown_connections 2024-03-29T11:02:48.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:48.103+0000 7febfb59e640 1 -- 172.21.15.71:0/2782507331 wait complete. 2024-03-29T11:02:48.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:48.103+0000 7febfb59e640 1 librados: shutdown 2024-03-29T11:02:48.098 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:54.906 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:55.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.069+0000 7f38d29d6640 10 monclient: get_monmap_and_config 2024-03-29T11:02:55.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.069+0000 7f38d29d6640 10 monclient: build_initial_monmap 2024-03-29T11:02:55.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.069+0000 7f38d29d6640 10 monclient: monmap: 2024-03-29T11:02:55.049 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:02:55.050 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:55.050 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:02:55.070678+0000 2024-03-29T11:02:55.050 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:02:55.070678+0000 2024-03-29T11:02:55.050 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:55.050 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:55.050 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:02:55.050 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:02:55.050 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:02:55.050 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:55.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.071+0000 7f38d29d6640 1 Processor -- start 2024-03-29T11:02:55.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.071+0000 7f38d29d6640 1 -- start start 2024-03-29T11:02:55.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.071+0000 7f38d29d6640 10 monclient: init 2024-03-29T11:02:55.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.072+0000 7f38d29d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:55.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.072+0000 7f38d29d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:02:55.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.072+0000 7f38d29d6640 10 monclient(hunting): picked mon.noname-b con 0x7f38cc0b0500 addr v1:172.21.15.71:6790/0 2024-03-29T11:02:55.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.072+0000 7f38d29d6640 10 monclient(hunting): picked mon.noname-c con 0x7f38cc0b08e0 addr v1:172.21.15.137:6789/0 2024-03-29T11:02:55.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.072+0000 7f38d29d6640 10 monclient(hunting): picked mon.noname-a con 0x7f38cc0aeca0 addr v1:172.21.15.71:6789/0 2024-03-29T11:02:55.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.072+0000 7f38cb7fe640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f38cc0b08e0 0x7f38cc13d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:43556/0 (socket says 172.21.15.71:43556) 2024-03-29T11:02:55.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.072+0000 7f38d29d6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38cc140620 con 0x7f38cc0aeca0 2024-03-29T11:02:55.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.072+0000 7f38d29d6640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38cc13daf0 con 0x7f38cc0b08e0 2024-03-29T11:02:55.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.072+0000 7f38d29d6640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38cc0ae180 con 0x7f38cc0b0500 2024-03-29T11:02:55.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.072+0000 7f38d29d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:55.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.072+0000 7f38d29d6640 10 monclient(hunting): authenticate will time out at 931.331019s 2024-03-29T11:02:55.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.072+0000 7f38d0f73640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f38cc0aeca0 0x7f38cc14b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46388/0 (socket says 172.21.15.71:46388) 2024-03-29T11:02:55.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.073+0000 7f38d0f73640 1 -- 172.21.15.71:0/1439156316 learned_addr learned my addr 172.21.15.71:0/1439156316 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:02:55.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.073+0000 7f38caffd640 1 -- 172.21.15.71:0/1439156316 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2883070255 0 0) 0x7f38cc13daf0 con 0x7f38cc0b08e0 2024-03-29T11:02:55.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.073+0000 7f38caffd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:55.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.073+0000 7f38caffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:55.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.073+0000 7f38caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:55.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.073+0000 7f38caffd640 10 monclient(hunting): my global_id is 4297 2024-03-29T11:02:55.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.073+0000 7f38caffd640 1 -- 172.21.15.71:0/1439156316 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38b8003520 con 0x7f38cc0b08e0 2024-03-29T11:02:55.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.074+0000 7f38caffd640 1 -- 172.21.15.71:0/1439156316 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1702484555 0 0) 0x7f38cc140620 con 0x7f38cc0aeca0 2024-03-29T11:02:55.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.074+0000 7f38caffd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:55.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.074+0000 7f38caffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:55.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.074+0000 7f38caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:55.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.074+0000 7f38caffd640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:02:55.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.074+0000 7f38caffd640 1 -- 172.21.15.71:0/1439156316 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38cc13daf0 con 0x7f38cc0aeca0 2024-03-29T11:02:55.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.074+0000 7f38caffd640 1 -- 172.21.15.71:0/1439156316 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756069814 0 0) 0x7f38cc0ae180 con 0x7f38cc0b0500 2024-03-29T11:02:55.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.074+0000 7f38caffd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:55.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.074+0000 7f38caffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:55.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.074+0000 7f38caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:55.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.074+0000 7f38caffd640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:02:55.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.074+0000 7f38caffd640 1 -- 172.21.15.71:0/1439156316 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38cc140620 con 0x7f38cc0b0500 2024-03-29T11:02:55.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.074+0000 7f38caffd640 1 -- 172.21.15.71:0/1439156316 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1186013201 0 0) 0x7f38b8003520 con 0x7f38cc0b08e0 2024-03-29T11:02:55.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.074+0000 7f38caffd640 1 -- 172.21.15.71:0/1439156316 >> v1:172.21.15.71:6790/0 conn(0x7f38cc0b0500 legacy=0x7f38cc13ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:55.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.074+0000 7f38caffd640 1 -- 172.21.15.71:0/1439156316 >> v1:172.21.15.71:6789/0 conn(0x7f38cc0aeca0 legacy=0x7f38cc14b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:55.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.074+0000 7f38caffd640 10 monclient: _finish_hunting 0 2024-03-29T11:02:55.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.074+0000 7f38caffd640 1 monclient: found mon.noname-c 2024-03-29T11:02:55.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.074+0000 7f38caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:55.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.074+0000 7f38caffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.137:6789/0 2024-03-29T11:02:55.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.074+0000 7f38caffd640 1 -- 172.21.15.71:0/1439156316 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38cc0accf0 con 0x7f38cc0b08e0 2024-03-29T11:02:55.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.074+0000 7f38caffd640 10 monclient: _finish_auth 0 2024-03-29T11:02:55.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.074+0000 7f38caffd640 10 monclient: _check_auth_tickets 2024-03-29T11:02:55.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.074+0000 7f38caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:55.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.074+0000 7f38caffd640 1 -- 172.21.15.71:0/1439156316 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f38b4001610 con 0x7f38cc0b08e0 2024-03-29T11:02:55.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.074+0000 7f38caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:55.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.074+0000 7f38caffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:55.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.074+0000 7f38caffd640 10 monclient: dump: 2024-03-29T11:02:55.061 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:55.061 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:55.061 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:55.061 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:55.061 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:55.061 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:55.061 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:55.061 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:55.062 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:55.062 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:55.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.075+0000 7f38caffd640 1 -- 172.21.15.71:0/1439156316 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38b4001770 con 0x7f38cc0b08e0 2024-03-29T11:02:55.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.075+0000 7f38caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:55.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.075+0000 7f38caffd640 1 -- 172.21.15.71:0/1439156316 <== mon.1 v1:172.21.15.137:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f38b4001bc0 con 0x7f38cc0b08e0 2024-03-29T11:02:55.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.075+0000 7f38caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:55.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.075+0000 7f38caffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:02:55.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.075+0000 7f38caffd640 10 monclient: dump: 2024-03-29T11:02:55.062 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:55.062 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:55.062 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:55.062 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:55.062 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:55.063 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:55.063 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:55.063 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:55.063 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:55.063 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:55.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.075+0000 7f38d29d6640 5 monclient: authenticate success, global_id 4297 2024-03-29T11:02:55.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.075+0000 7f38d29d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:55.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.075+0000 7f38d29d6640 4 set_mon_vals no callback set 2024-03-29T11:02:55.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.076+0000 7f38d29d6640 10 monclient: shutdown 2024-03-29T11:02:55.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.076+0000 7f38d29d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:55.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.076+0000 7f38d29d6640 1 -- 172.21.15.71:0/1439156316 >> v1:172.21.15.137:6789/0 conn(0x7f38cc0b08e0 legacy=0x7f38cc13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:55.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.076+0000 7f38d29d6640 1 -- 172.21.15.71:0/1439156316 shutdown_connections 2024-03-29T11:02:55.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.076+0000 7f38d29d6640 1 -- 172.21.15.71:0/1439156316 >> 172.21.15.71:0/1439156316 conn(0x7f38cc06b660 msgr2=0x7f38cc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:55.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.076+0000 7f38d29d6640 1 -- 172.21.15.71:0/1439156316 shutdown_connections 2024-03-29T11:02:55.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.076+0000 7f38d29d6640 1 -- 172.21.15.71:0/1439156316 wait complete. 2024-03-29T11:02:55.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.078+0000 7f38d29d6640 10 monclient: build_initial_monmap 2024-03-29T11:02:55.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.078+0000 7f38d29d6640 10 monclient: monmap: 2024-03-29T11:02:55.064 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:02:55.064 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:55.064 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:02:55.064 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:02:55.064 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:55.065 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:55.065 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:02:55.065 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:02:55.065 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:02:55.065 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:55.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.078+0000 7f38d29d6640 1 librados: starting msgr at 2024-03-29T11:02:55.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.078+0000 7f38d29d6640 1 librados: starting objecter 2024-03-29T11:02:55.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.079+0000 7f38d29d6640 1 Processor -- start 2024-03-29T11:02:55.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.079+0000 7f38d29d6640 1 -- start start 2024-03-29T11:02:55.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.079+0000 7f38d29d6640 1 librados: setting wanted keys 2024-03-29T11:02:55.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.079+0000 7f38d29d6640 1 librados: calling monclient init 2024-03-29T11:02:55.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.079+0000 7f38d29d6640 10 monclient: init 2024-03-29T11:02:55.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.080+0000 7f38d29d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:55.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.080+0000 7f38d29d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:02:55.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.080+0000 7f38d29d6640 10 monclient(hunting): picked mon.noname-c con 0x7f38cc0aeca0 addr v1:172.21.15.71:6790/0 2024-03-29T11:02:55.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.080+0000 7f38d29d6640 10 monclient(hunting): picked mon.noname-a con 0x7f38cc0b0500 addr v1:172.21.15.71:6789/0 2024-03-29T11:02:55.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.080+0000 7f38d29d6640 10 monclient(hunting): picked mon.noname-b con 0x7f38cc13c240 addr v1:172.21.15.137:6789/0 2024-03-29T11:02:55.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38d0f73640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f38cc13c240 0x7f38cc1cbac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:44328/0 (socket says 172.21.15.71:44328) 2024-03-29T11:02:55.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38d0f73640 1 -- 172.21.15.71:0/3908594704 learned_addr learned my addr 172.21.15.71:0/3908594704 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:02:55.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38cbfff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f38cc0aeca0 0x7f38cc1d8da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:53144/0 (socket says 172.21.15.71:53144) 2024-03-29T11:02:55.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38d29d6640 1 -- 172.21.15.71:0/3908594704 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38cc1d98b0 con 0x7f38cc0b0500 2024-03-29T11:02:55.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38d29d6640 1 -- 172.21.15.71:0/3908594704 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38cc11bad0 con 0x7f38cc13c240 2024-03-29T11:02:55.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38d29d6640 1 -- 172.21.15.71:0/3908594704 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38cc1d94b0 con 0x7f38cc0aeca0 2024-03-29T11:02:55.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38d29d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:55.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38d29d6640 10 monclient(hunting): authenticate will time out at 931.339467s 2024-03-29T11:02:55.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38c8ff9640 10 client.?.objecter ms_handle_connect 0x7f38cc13c240 2024-03-29T11:02:55.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:55.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38c8ff9640 10 client.?.objecter ms_handle_connect 0x7f38cc0aeca0 2024-03-29T11:02:55.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:55.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38c8ff9640 10 client.?.objecter ms_handle_connect 0x7f38cc0b0500 2024-03-29T11:02:55.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:55.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38c8ff9640 1 -- 172.21.15.71:0/3908594704 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 142428137 0 0) 0x7f38cc1d94b0 con 0x7f38cc0aeca0 2024-03-29T11:02:55.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38c8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:02:55.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:55.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:55.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38c8ff9640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:02:55.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38c8ff9640 1 -- 172.21.15.71:0/3908594704 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38ac004eb0 con 0x7f38cc0aeca0 2024-03-29T11:02:55.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38c8ff9640 1 -- 172.21.15.71:0/3908594704 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3166187226 0 0) 0x7f38cc11bad0 con 0x7f38cc13c240 2024-03-29T11:02:55.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38c8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:02:55.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:55.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:55.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38c8ff9640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:02:55.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38c8ff9640 1 -- 172.21.15.71:0/3908594704 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38cc1d94b0 con 0x7f38cc13c240 2024-03-29T11:02:55.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38c8ff9640 1 -- 172.21.15.71:0/3908594704 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2229292522 0 0) 0x7f38cc1d98b0 con 0x7f38cc0b0500 2024-03-29T11:02:55.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38c8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:02:55.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:55.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:55.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38c8ff9640 10 monclient(hunting): my global_id is 4332 2024-03-29T11:02:55.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.081+0000 7f38c8ff9640 1 -- 172.21.15.71:0/3908594704 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38cc11bad0 con 0x7f38cc0b0500 2024-03-29T11:02:55.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38c8ff9640 1 -- 172.21.15.71:0/3908594704 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3462571434 0 0) 0x7f38ac004eb0 con 0x7f38cc0aeca0 2024-03-29T11:02:55.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38c8ff9640 1 -- 172.21.15.71:0/3908594704 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38cc1d98b0 con 0x7f38cc0aeca0 2024-03-29T11:02:55.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38c8ff9640 1 -- 172.21.15.71:0/3908594704 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f38c40018f0 con 0x7f38cc0aeca0 2024-03-29T11:02:55.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:55.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38c8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:55.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38c8ff9640 10 monclient(hunting): dump: 2024-03-29T11:02:55.083 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:55.083 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:55.083 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:55.083 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:55.083 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:55.084 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:55.084 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:55.084 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:55.084 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:55.084 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:55.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38c8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:55.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38c8ff9640 1 -- 172.21.15.71:0/3908594704 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2014729809 0 0) 0x7f38cc11bad0 con 0x7f38cc0b0500 2024-03-29T11:02:55.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38c8ff9640 1 -- 172.21.15.71:0/3908594704 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38ac004eb0 con 0x7f38cc0b0500 2024-03-29T11:02:55.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38c8ff9640 1 -- 172.21.15.71:0/3908594704 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f38b4001a80 con 0x7f38cc0b0500 2024-03-29T11:02:55.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:55.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38c8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:55.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38c8ff9640 10 monclient(hunting): dump: 2024-03-29T11:02:55.086 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:55.086 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:55.086 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:55.086 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:55.086 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:55.086 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:55.087 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:55.087 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:55.087 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:55.087 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:55.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38c8ff9640 1 -- 172.21.15.71:0/3908594704 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2702052926 0 0) 0x7f38cc1d98b0 con 0x7f38cc0aeca0 2024-03-29T11:02:55.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38c8ff9640 1 -- 172.21.15.71:0/3908594704 >> v1:172.21.15.137:6789/0 conn(0x7f38cc13c240 legacy=0x7f38cc1cbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:55.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38c8ff9640 1 -- 172.21.15.71:0/3908594704 >> v1:172.21.15.71:6789/0 conn(0x7f38cc0b0500 legacy=0x7f38cc1cb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:55.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38c8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:02:55.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38c8ff9640 1 monclient: found mon.c 2024-03-29T11:02:55.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38c8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:55.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38c8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:02:55.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38c8ff9640 1 -- 172.21.15.71:0/3908594704 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38cc1e4120 con 0x7f38cc0aeca0 2024-03-29T11:02:55.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38c8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:02:55.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38c8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:02:55.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:55.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38d29d6640 5 monclient: authenticate success, global_id 4319 2024-03-29T11:02:55.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38d29d6640 10 monclient: _renew_subs 2024-03-29T11:02:55.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38d29d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:02:55.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38d29d6640 1 -- 172.21.15.71:0/3908594704 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38cc0aaae0 con 0x7f38cc0aeca0 2024-03-29T11:02:55.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38d29d6640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:55.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38d29d6640 10 monclient: _renew_subs 2024-03-29T11:02:55.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38d29d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:02:55.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38d29d6640 1 -- 172.21.15.71:0/3908594704 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38cc0aaf10 con 0x7f38cc0aeca0 2024-03-29T11:02:55.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.082+0000 7f38d29d6640 1 librados: init done 2024-03-29T11:02:55.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.085+0000 7f38d29d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:55.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.085+0000 7f38d29d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:55.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.085+0000 7f38d29d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:02:55.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.085+0000 7f38c8ff9640 1 -- 172.21.15.71:0/3908594704 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38c4003610 con 0x7f38cc0aeca0 2024-03-29T11:02:55.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.085+0000 7f38d29d6640 1 -- 172.21.15.71:0/3908594704 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38cc0ad0c0 con 0x7f38cc0aeca0 2024-03-29T11:02:55.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.085+0000 7f38c8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:55.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.085+0000 7f38c8ff9640 1 -- 172.21.15.71:0/3908594704 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f38c40039a0 con 0x7f38cc0aeca0 2024-03-29T11:02:55.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.085+0000 7f38c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:55.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.085+0000 7f38c8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:55.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.085+0000 7f38c8ff9640 10 monclient: dump: 2024-03-29T11:02:55.092 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:02:55.092 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:02:55.092 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:55.092 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:02:55.092 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:55.093 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:02:55.093 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:02:55.093 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:02:55.093 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:02:55.093 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:02:55.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.085+0000 7f38ca7fc640 4 set_mon_vals no callback set 2024-03-29T11:02:55.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.087+0000 7f38c8ff9640 1 -- 172.21.15.71:0/3908594704 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f38c4003bc0 con 0x7f38cc0aeca0 2024-03-29T11:02:55.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.088+0000 7f38c8ff9640 1 -- 172.21.15.71:0/3908594704 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f38c4084850 con 0x7f38cc0aeca0 2024-03-29T11:02:55.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.088+0000 7f38c8ff9640 10 client.4319.objecter ms_dispatch 0x7f38cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:02:55.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.088+0000 7f38cb7fe640 10 monclient: get_auth_request con 0x7f38ac0723a0 auth_method 0 2024-03-29T11:02:55.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.088+0000 7f38c8ff9640 3 client.4319.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:02:55.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.088+0000 7f38c8ff9640 3 client.4319.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:02:55.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.088+0000 7f38c8ff9640 20 client.4319.objecter dump_active .. 0 homeless 2024-03-29T11:02:55.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.088+0000 7f38c8ff9640 10 client.4319.objecter ms_handle_connect 0x7f38ac0723a0 2024-03-29T11:02:55.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.096+0000 7f38c8ff9640 1 -- 172.21.15.71:0/3908594704 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f38c4050130 con 0x7f38cc0aeca0 2024-03-29T11:02:55.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.096+0000 7f38c8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:55.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.096+0000 7f38c8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:55.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.391+0000 7f38d29d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:55.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.391+0000 7f38d29d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:55.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.391+0000 7f38d29d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:02:55.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.391+0000 7f38d29d6640 1 -- 172.21.15.71:0/3908594704 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38cc1d9640 con 0x7f38cc0aeca0 2024-03-29T11:02:55.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.391+0000 7f38c8ff9640 1 -- 172.21.15.71:0/3908594704 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f38c4052060 con 0x7f38cc0aeca0 2024-03-29T11:02:55.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.392+0000 7f38c8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:55.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.392+0000 7f38c8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:55.373 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:02:55.373 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:55.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.395+0000 7f38d29d6640 10 librados: watch_flush enter 2024-03-29T11:02:55.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.395+0000 7f38d29d6640 10 librados: watch_flush exit 2024-03-29T11:02:55.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.395+0000 7f38d29d6640 20 client.4319.objecter shutdown clearing up homeless session... 2024-03-29T11:02:55.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.395+0000 7f38d29d6640 10 client.4319.objecter successfully canceled tick 2024-03-29T11:02:55.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.395+0000 7f38d29d6640 1 -- 172.21.15.71:0/3908594704 >> v1:172.21.15.137:6816/973332665 conn(0x7f38ac0723a0 legacy=0x7f38ac074860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:55.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.395+0000 7f38d29d6640 10 monclient: shutdown 2024-03-29T11:02:55.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.395+0000 7f38d29d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:55.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.395+0000 7f38d29d6640 1 -- 172.21.15.71:0/3908594704 >> v1:172.21.15.71:6790/0 conn(0x7f38cc0aeca0 legacy=0x7f38cc1d8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:55.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.395+0000 7f38d29d6640 1 -- 172.21.15.71:0/3908594704 shutdown_connections 2024-03-29T11:02:55.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.395+0000 7f38d29d6640 1 -- 172.21.15.71:0/3908594704 >> 172.21.15.71:0/3908594704 conn(0x7f38cc06b660 msgr2=0x7f38cc06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:55.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.395+0000 7f38d29d6640 1 -- 172.21.15.71:0/3908594704 shutdown_connections 2024-03-29T11:02:55.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.396+0000 7f38d29d6640 1 -- 172.21.15.71:0/3908594704 wait complete. 2024-03-29T11:02:55.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:02:55.396+0000 7f38d29d6640 1 librados: shutdown 2024-03-29T11:02:55.389 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:03:02.196 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:03:02.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.367+0000 7fdb96f3a640 10 monclient: get_monmap_and_config 2024-03-29T11:03:02.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.367+0000 7fdb96f3a640 10 monclient: build_initial_monmap 2024-03-29T11:03:02.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.367+0000 7fdb96f3a640 10 monclient: monmap: 2024-03-29T11:03:02.349 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:03:02.349 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:02.349 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:03:02.368562+0000 2024-03-29T11:03:02.349 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:03:02.368562+0000 2024-03-29T11:03:02.349 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:02.349 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:02.349 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:03:02.349 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:03:02.349 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:03:02.350 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:02.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.368+0000 7fdb96f3a640 1 Processor -- start 2024-03-29T11:03:02.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.369+0000 7fdb96f3a640 1 -- start start 2024-03-29T11:03:02.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.369+0000 7fdb96f3a640 10 monclient: init 2024-03-29T11:03:02.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.370+0000 7fdb96f3a640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:02.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.370+0000 7fdb96f3a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:03:02.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.370+0000 7fdb96f3a640 10 monclient(hunting): picked mon.noname-c con 0x7fdb900a9ac0 addr v1:172.21.15.137:6789/0 2024-03-29T11:03:02.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.370+0000 7fdb96f3a640 10 monclient(hunting): picked mon.noname-b con 0x7fdb9013ee90 addr v1:172.21.15.71:6790/0 2024-03-29T11:03:02.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.370+0000 7fdb96f3a640 10 monclient(hunting): picked mon.noname-a con 0x7fdb900af900 addr v1:172.21.15.71:6789/0 2024-03-29T11:03:02.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.370+0000 7fdb96f3a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb900afd10 con 0x7fdb900af900 2024-03-29T11:03:02.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.370+0000 7fdb96f3a640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb900af6f0 con 0x7fdb900a9ac0 2024-03-29T11:03:02.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.370+0000 7fdb96f3a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb900aac80 con 0x7fdb9013ee90 2024-03-29T11:03:02.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.370+0000 7fdb96f3a640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:02.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.370+0000 7fdb96f3a640 10 monclient(hunting): authenticate will time out at 938.628512s 2024-03-29T11:03:02.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.374+0000 7fdb954d7640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fdb900af900 0x7fdb90149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:39151/0 (socket says 172.21.15.71:39151) 2024-03-29T11:03:02.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.374+0000 7fdb954d7640 1 -- 172.21.15.71:0/2108619262 learned_addr learned my addr 172.21.15.71:0/2108619262 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:03:02.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.374+0000 7fdb8f7fe640 1 -- 172.21.15.71:0/2108619262 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 142533014 0 0) 0x7fdb900afd10 con 0x7fdb900af900 2024-03-29T11:03:02.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.374+0000 7fdb8f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:03:02.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.374+0000 7fdb8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:02.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.374+0000 7fdb8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:02.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.374+0000 7fdb8f7fe640 10 monclient(hunting): my global_id is 4335 2024-03-29T11:03:02.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.374+0000 7fdb8f7fe640 1 -- 172.21.15.71:0/2108619262 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb78003520 con 0x7fdb900af900 2024-03-29T11:03:02.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.375+0000 7fdb8f7fe640 1 -- 172.21.15.71:0/2108619262 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 688817561 0 0) 0x7fdb900aac80 con 0x7fdb9013ee90 2024-03-29T11:03:02.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.375+0000 7fdb8f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:03:02.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.375+0000 7fdb8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:02.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.375+0000 7fdb8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:02.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.375+0000 7fdb8f7fe640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:03:02.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.375+0000 7fdb8f7fe640 1 -- 172.21.15.71:0/2108619262 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb900afd10 con 0x7fdb9013ee90 2024-03-29T11:03:02.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.376+0000 7fdb8f7fe640 1 -- 172.21.15.71:0/2108619262 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1469248245 0 0) 0x7fdb900afd10 con 0x7fdb9013ee90 2024-03-29T11:03:02.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.376+0000 7fdb8f7fe640 1 -- 172.21.15.71:0/2108619262 >> v1:172.21.15.137:6789/0 conn(0x7fdb900a9ac0 legacy=0x7fdb900adfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:03:02.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.376+0000 7fdb8f7fe640 1 -- 172.21.15.71:0/2108619262 >> v1:172.21.15.71:6789/0 conn(0x7fdb900af900 legacy=0x7fdb90149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:02.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.376+0000 7fdb8f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:03:02.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.376+0000 7fdb8f7fe640 1 monclient: found mon.noname-b 2024-03-29T11:03:02.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.376+0000 7fdb8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:02.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.376+0000 7fdb8f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:03:02.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.376+0000 7fdb8f7fe640 1 -- 172.21.15.71:0/2108619262 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb900aaf60 con 0x7fdb9013ee90 2024-03-29T11:03:02.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.376+0000 7fdb8f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:03:02.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.376+0000 7fdb8f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:03:02.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.376+0000 7fdb8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:02.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.376+0000 7fdb8f7fe640 1 -- 172.21.15.71:0/2108619262 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fdb80001670 con 0x7fdb9013ee90 2024-03-29T11:03:02.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.376+0000 7fdb8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:02.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.376+0000 7fdb8f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:03:02.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.376+0000 7fdb8f7fe640 10 monclient: dump: 2024-03-29T11:03:02.358 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:02.358 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:02.358 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:02.358 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:02.358 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:02.358 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:02.358 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:02.358 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:02.358 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:02.358 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:02.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.376+0000 7fdb96f3a640 5 monclient: authenticate success, global_id 4322 2024-03-29T11:03:02.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.376+0000 7fdb96f3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:02.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.377+0000 7fdb8f7fe640 1 -- 172.21.15.71:0/2108619262 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb80001890 con 0x7fdb9013ee90 2024-03-29T11:03:02.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.377+0000 7fdb8f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:02.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.377+0000 7fdb8f7fe640 1 -- 172.21.15.71:0/2108619262 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fdb80001c20 con 0x7fdb9013ee90 2024-03-29T11:03:02.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.377+0000 7fdb96f3a640 10 monclient: get_monmap_and_config success 2024-03-29T11:03:02.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.377+0000 7fdb96f3a640 4 set_mon_vals no callback set 2024-03-29T11:03:02.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.377+0000 7fdb8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:02.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.377+0000 7fdb8f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:02.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.377+0000 7fdb8f7fe640 10 monclient: dump: 2024-03-29T11:03:02.361 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:02.361 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:02.361 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:02.361 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:02.361 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:02.361 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:02.361 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:02.362 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:02.362 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:02.362 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:02.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.378+0000 7fdb96f3a640 10 monclient: shutdown 2024-03-29T11:03:02.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.378+0000 7fdb96f3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:02.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.378+0000 7fdb96f3a640 1 -- 172.21.15.71:0/2108619262 >> v1:172.21.15.71:6790/0 conn(0x7fdb9013ee90 legacy=0x7fdb9013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:02.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.378+0000 7fdb96f3a640 1 -- 172.21.15.71:0/2108619262 shutdown_connections 2024-03-29T11:03:02.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.378+0000 7fdb96f3a640 1 -- 172.21.15.71:0/2108619262 >> 172.21.15.71:0/2108619262 conn(0x7fdb900b35e0 msgr2=0x7fdb900b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:02.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.378+0000 7fdb96f3a640 1 -- 172.21.15.71:0/2108619262 shutdown_connections 2024-03-29T11:03:02.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.378+0000 7fdb96f3a640 1 -- 172.21.15.71:0/2108619262 wait complete. 2024-03-29T11:03:02.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.379+0000 7fdb96f3a640 10 monclient: build_initial_monmap 2024-03-29T11:03:02.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.379+0000 7fdb96f3a640 10 monclient: monmap: 2024-03-29T11:03:02.363 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:03:02.363 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:03:02.363 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:03:02.363 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:03:02.363 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:02.363 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:02.363 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:03:02.363 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:03:02.363 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:03:02.363 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:02.364 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.380+0000 7fdb96f3a640 1 librados: starting msgr at 2024-03-29T11:03:02.364 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.380+0000 7fdb96f3a640 1 librados: starting objecter 2024-03-29T11:03:02.364 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.380+0000 7fdb96f3a640 1 Processor -- start 2024-03-29T11:03:02.364 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.380+0000 7fdb96f3a640 1 -- start start 2024-03-29T11:03:02.364 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.380+0000 7fdb96f3a640 1 librados: setting wanted keys 2024-03-29T11:03:02.364 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.380+0000 7fdb96f3a640 1 librados: calling monclient init 2024-03-29T11:03:02.364 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.380+0000 7fdb96f3a640 10 monclient: init 2024-03-29T11:03:02.364 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.381+0000 7fdb96f3a640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:02.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.382+0000 7fdb96f3a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:03:02.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.382+0000 7fdb96f3a640 10 monclient(hunting): picked mon.noname-a con 0x7fdb900a9ac0 addr v1:172.21.15.71:6789/0 2024-03-29T11:03:02.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.382+0000 7fdb96f3a640 10 monclient(hunting): picked mon.noname-c con 0x7fdb900af900 addr v1:172.21.15.71:6790/0 2024-03-29T11:03:02.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.382+0000 7fdb96f3a640 10 monclient(hunting): picked mon.noname-b con 0x7fdb9013ee90 addr v1:172.21.15.137:6789/0 2024-03-29T11:03:02.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.382+0000 7fdb96f3a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb9011d140 con 0x7fdb900a9ac0 2024-03-29T11:03:02.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.382+0000 7fdb96f3a640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb901228d0 con 0x7fdb9013ee90 2024-03-29T11:03:02.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.382+0000 7fdb96f3a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb9011d390 con 0x7fdb900af900 2024-03-29T11:03:02.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.382+0000 7fdb96f3a640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:02.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.382+0000 7fdb96f3a640 10 monclient(hunting): authenticate will time out at 938.640884s 2024-03-29T11:03:02.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.385+0000 7fdb8ffff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fdb900af900 0x7fdb9011b1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:42787/0 (socket says 172.21.15.71:42787) 2024-03-29T11:03:02.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.385+0000 7fdb8ffff640 1 -- 172.21.15.71:0/2875378685 learned_addr learned my addr 172.21.15.71:0/2875378685 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:03:02.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.385+0000 7fdb8d7fa640 10 client.?.objecter ms_handle_connect 0x7fdb900af900 2024-03-29T11:03:02.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.386+0000 7fdb8d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:02.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.386+0000 7fdb8d7fa640 10 client.?.objecter ms_handle_connect 0x7fdb900a9ac0 2024-03-29T11:03:02.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.386+0000 7fdb8d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:02.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.386+0000 7fdb8d7fa640 1 -- 172.21.15.71:0/2875378685 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 329363498 0 0) 0x7fdb9011d140 con 0x7fdb900a9ac0 2024-03-29T11:03:02.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.386+0000 7fdb8d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:03:02.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.386+0000 7fdb8d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:02.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.386+0000 7fdb8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:02.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.386+0000 7fdb8d7fa640 10 monclient(hunting): my global_id is 4338 2024-03-29T11:03:02.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.386+0000 7fdb8d7fa640 1 -- 172.21.15.71:0/2875378685 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb7c004eb0 con 0x7fdb900a9ac0 2024-03-29T11:03:02.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.386+0000 7fdb8d7fa640 1 -- 172.21.15.71:0/2875378685 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 313047613 0 0) 0x7fdb9011d390 con 0x7fdb900af900 2024-03-29T11:03:02.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.386+0000 7fdb8d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:03:02.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.386+0000 7fdb8d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:02.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.386+0000 7fdb8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:02.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.386+0000 7fdb8d7fa640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:03:02.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.386+0000 7fdb8d7fa640 1 -- 172.21.15.71:0/2875378685 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb9011d140 con 0x7fdb900af900 2024-03-29T11:03:02.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.387+0000 7fdb8d7fa640 1 -- 172.21.15.71:0/2875378685 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2141894757 0 0) 0x7fdb7c004eb0 con 0x7fdb900a9ac0 2024-03-29T11:03:02.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.387+0000 7fdb8d7fa640 1 -- 172.21.15.71:0/2875378685 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb9011d390 con 0x7fdb900a9ac0 2024-03-29T11:03:02.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.387+0000 7fdb8d7fa640 1 -- 172.21.15.71:0/2875378685 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fdb88001bf0 con 0x7fdb900a9ac0 2024-03-29T11:03:02.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.387+0000 7fdb8d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:02.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.387+0000 7fdb8d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:03:02.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.387+0000 7fdb8d7fa640 10 monclient(hunting): dump: 2024-03-29T11:03:02.373 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:02.373 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:02.373 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:02.374 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:02.374 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:02.374 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:02.374 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:02.374 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:02.374 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:02.374 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:02.374 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.387+0000 7fdb8d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:03:02.374 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.387+0000 7fdb8d7fa640 1 -- 172.21.15.71:0/2875378685 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3424862620 0 0) 0x7fdb9011d140 con 0x7fdb900af900 2024-03-29T11:03:02.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.387+0000 7fdb8d7fa640 1 -- 172.21.15.71:0/2875378685 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb7c004eb0 con 0x7fdb900af900 2024-03-29T11:03:02.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.387+0000 7fdb8d7fa640 1 -- 172.21.15.71:0/2875378685 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2457658736 0 0) 0x7fdb9011d390 con 0x7fdb900a9ac0 2024-03-29T11:03:02.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.387+0000 7fdb8d7fa640 1 -- 172.21.15.71:0/2875378685 >> v1:172.21.15.71:6790/0 conn(0x7fdb900af900 legacy=0x7fdb9011b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:02.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.388+0000 7fdb8d7fa640 1 -- 172.21.15.71:0/2875378685 >> v1:172.21.15.137:6789/0 conn(0x7fdb9013ee90 legacy=0x7fdb9011b8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:03:02.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.389+0000 7fdb8d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:03:02.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.389+0000 7fdb8d7fa640 1 monclient: found mon.a 2024-03-29T11:03:02.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.389+0000 7fdb8d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:02.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.389+0000 7fdb8d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:03:02.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.389+0000 7fdb8d7fa640 1 -- 172.21.15.71:0/2875378685 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb9012cf20 con 0x7fdb900a9ac0 2024-03-29T11:03:02.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.389+0000 7fdb8d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:03:02.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.389+0000 7fdb8d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:03:02.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.390+0000 7fdb8d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:02.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.390+0000 7fdb96f3a640 5 monclient: authenticate success, global_id 4338 2024-03-29T11:03:02.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.390+0000 7fdb96f3a640 10 monclient: _renew_subs 2024-03-29T11:03:02.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.390+0000 7fdb96f3a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:03:02.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.390+0000 7fdb96f3a640 1 -- 172.21.15.71:0/2875378685 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb9012d170 con 0x7fdb900a9ac0 2024-03-29T11:03:02.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.390+0000 7fdb96f3a640 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:03:02.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.390+0000 7fdb96f3a640 10 monclient: _renew_subs 2024-03-29T11:03:02.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.390+0000 7fdb96f3a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:03:02.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.390+0000 7fdb96f3a640 1 -- 172.21.15.71:0/2875378685 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb900725b0 con 0x7fdb900a9ac0 2024-03-29T11:03:02.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.390+0000 7fdb96f3a640 1 librados: init done 2024-03-29T11:03:02.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.392+0000 7fdb76ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:02.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.392+0000 7fdb76ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:02.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.392+0000 7fdb76ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:03:02.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.392+0000 7fdb76ffd640 1 -- 172.21.15.71:0/2875378685 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb9006dfd0 con 0x7fdb900a9ac0 2024-03-29T11:03:02.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.393+0000 7fdb8d7fa640 1 -- 172.21.15.71:0/2875378685 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb88002100 con 0x7fdb900a9ac0 2024-03-29T11:03:02.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.393+0000 7fdb8d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:02.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.393+0000 7fdb8d7fa640 1 -- 172.21.15.71:0/2875378685 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fdb88003bb0 con 0x7fdb900a9ac0 2024-03-29T11:03:02.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.393+0000 7fdb8d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:02.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.393+0000 7fdb8d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:03:02.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.393+0000 7fdb8d7fa640 10 monclient: dump: 2024-03-29T11:03:02.379 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:02.379 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:02.379 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:02.379 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:02.379 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:02.379 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:02.379 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:02.379 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:02.379 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:02.379 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:02.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.393+0000 7fdb8effd640 4 set_mon_vals no callback set 2024-03-29T11:03:02.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.395+0000 7fdb8d7fa640 1 -- 172.21.15.71:0/2875378685 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7fdb88019b80 con 0x7fdb900a9ac0 2024-03-29T11:03:02.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.395+0000 7fdb8d7fa640 1 -- 172.21.15.71:0/2875378685 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7fdb88083ab0 con 0x7fdb900a9ac0 2024-03-29T11:03:02.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.395+0000 7fdb8ffff640 10 monclient: get_auth_request con 0x7fdb7c06ddc0 auth_method 0 2024-03-29T11:03:02.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.395+0000 7fdb8d7fa640 10 client.4338.objecter ms_dispatch 0x7fdb90000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:03:02.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.396+0000 7fdb8d7fa640 3 client.4338.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:03:02.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.396+0000 7fdb8d7fa640 3 client.4338.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:03:02.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.396+0000 7fdb8d7fa640 20 client.4338.objecter dump_active .. 0 homeless 2024-03-29T11:03:02.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.396+0000 7fdb8d7fa640 10 client.4338.objecter ms_handle_connect 0x7fdb7c06ddc0 2024-03-29T11:03:02.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.404+0000 7fdb8d7fa640 1 -- 172.21.15.71:0/2875378685 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb8804f390 con 0x7fdb900a9ac0 2024-03-29T11:03:02.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.404+0000 7fdb8d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:02.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.404+0000 7fdb8d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:03:02.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.722+0000 7fdb76ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:02.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.722+0000 7fdb76ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:02.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.722+0000 7fdb76ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:03:02.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.722+0000 7fdb76ffd640 1 -- 172.21.15.71:0/2875378685 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb90075f10 con 0x7fdb900a9ac0 2024-03-29T11:03:02.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.722+0000 7fdb8d7fa640 1 -- 172.21.15.71:0/2875378685 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7fdb880512c0 con 0x7fdb900a9ac0 2024-03-29T11:03:02.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.722+0000 7fdb8d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:02.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.722+0000 7fdb8d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:03:02.703 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:03:02.703 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:03:02.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.727+0000 7fdb76ffd640 10 librados: watch_flush enter 2024-03-29T11:03:02.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.728+0000 7fdb76ffd640 10 librados: watch_flush exit 2024-03-29T11:03:02.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.728+0000 7fdb76ffd640 20 client.4338.objecter shutdown clearing up homeless session... 2024-03-29T11:03:02.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.728+0000 7fdb76ffd640 10 client.4338.objecter successfully canceled tick 2024-03-29T11:03:02.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.728+0000 7fdb76ffd640 1 -- 172.21.15.71:0/2875378685 >> v1:172.21.15.137:6816/973332665 conn(0x7fdb7c06ddc0 legacy=0x7fdb7c070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:02.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.728+0000 7fdb76ffd640 10 monclient: shutdown 2024-03-29T11:03:02.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.728+0000 7fdb76ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:02.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.728+0000 7fdb76ffd640 1 -- 172.21.15.71:0/2875378685 >> v1:172.21.15.71:6789/0 conn(0x7fdb900a9ac0 legacy=0x7fdb90070ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:02.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.729+0000 7fdb76ffd640 1 -- 172.21.15.71:0/2875378685 shutdown_connections 2024-03-29T11:03:02.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.729+0000 7fdb76ffd640 1 -- 172.21.15.71:0/2875378685 >> 172.21.15.71:0/2875378685 conn(0x7fdb900b35e0 msgr2=0x7fdb90130010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:02.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.729+0000 7fdb76ffd640 1 -- 172.21.15.71:0/2875378685 shutdown_connections 2024-03-29T11:03:02.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.729+0000 7fdb76ffd640 1 -- 172.21.15.71:0/2875378685 wait complete. 2024-03-29T11:03:02.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:02.730+0000 7fdb76ffd640 1 librados: shutdown 2024-03-29T11:03:02.728 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:03:09.534 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:03:09.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.707+0000 7f545c7ed640 10 monclient: get_monmap_and_config 2024-03-29T11:03:09.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.707+0000 7f545c7ed640 10 monclient: build_initial_monmap 2024-03-29T11:03:09.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.707+0000 7f545c7ed640 10 monclient: monmap: 2024-03-29T11:03:09.687 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:03:09.687 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:09.687 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:03:09.707928+0000 2024-03-29T11:03:09.687 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:03:09.707928+0000 2024-03-29T11:03:09.687 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:09.687 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:09.687 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:03:09.687 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:03:09.687 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:03:09.687 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:09.688 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.707+0000 7f545c7ed640 1 Processor -- start 2024-03-29T11:03:09.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.708+0000 7f545c7ed640 1 -- start start 2024-03-29T11:03:09.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.708+0000 7f545c7ed640 10 monclient: init 2024-03-29T11:03:09.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.709+0000 7f545c7ed640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:09.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.709+0000 7f545c7ed640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:03:09.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.709+0000 7f545c7ed640 10 monclient(hunting): picked mon.noname-c con 0x7f54540b46b0 addr v1:172.21.15.137:6789/0 2024-03-29T11:03:09.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.709+0000 7f545c7ed640 10 monclient(hunting): picked mon.noname-a con 0x7f54540b3da0 addr v1:172.21.15.71:6789/0 2024-03-29T11:03:09.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.709+0000 7f545c7ed640 10 monclient(hunting): picked mon.noname-b con 0x7f5454142a40 addr v1:172.21.15.71:6790/0 2024-03-29T11:03:09.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.709+0000 7f545c7ed640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54540b3530 con 0x7f54540b3da0 2024-03-29T11:03:09.691 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.709+0000 7f545c7ed640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54540b4280 con 0x7f54540b46b0 2024-03-29T11:03:09.691 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.709+0000 7f545c7ed640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54540aa9f0 con 0x7f5454142a40 2024-03-29T11:03:09.691 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.709+0000 7f545c7ed640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:09.691 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.709+0000 7f545c7ed640 10 monclient(hunting): authenticate will time out at 945.967291s 2024-03-29T11:03:09.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.713+0000 7f545ad8a640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f5454142a40 0x7f545414d870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:34775/0 (socket says 172.21.15.71:34775) 2024-03-29T11:03:09.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.713+0000 7f545ad8a640 1 -- 172.21.15.71:0/2738510123 learned_addr learned my addr 172.21.15.71:0/2738510123 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:03:09.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.714+0000 7f5459587640 1 -- 172.21.15.71:0/2738510123 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1484750596 0 0) 0x7f54540aa9f0 con 0x7f5454142a40 2024-03-29T11:03:09.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.714+0000 7f5459587640 10 monclient(hunting): _negotiate 2024-03-29T11:03:09.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.714+0000 7f5459587640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:09.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.714+0000 7f5459587640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:09.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.714+0000 7f5459587640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:03:09.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.714+0000 7f5459587640 1 -- 172.21.15.71:0/2738510123 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5440003520 con 0x7f5454142a40 2024-03-29T11:03:09.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.715+0000 7f5459587640 1 -- 172.21.15.71:0/2738510123 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3923768842 0 0) 0x7f5440003520 con 0x7f5454142a40 2024-03-29T11:03:09.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.715+0000 7f5459587640 1 -- 172.21.15.71:0/2738510123 >> v1:172.21.15.137:6789/0 conn(0x7f54540b46b0 legacy=0x7f5454142330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:03:09.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.715+0000 7f5459587640 1 -- 172.21.15.71:0/2738510123 >> v1:172.21.15.71:6789/0 conn(0x7f54540b3da0 legacy=0x7f545413b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:03:09.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.715+0000 7f5459587640 10 monclient: _finish_hunting 0 2024-03-29T11:03:09.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.715+0000 7f5459587640 1 monclient: found mon.noname-b 2024-03-29T11:03:09.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.715+0000 7f5459587640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:09.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.715+0000 7f5459587640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:03:09.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.715+0000 7f5459587640 1 -- 172.21.15.71:0/2738510123 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54540aacd0 con 0x7f5454142a40 2024-03-29T11:03:09.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.715+0000 7f5459587640 10 monclient: _finish_auth 0 2024-03-29T11:03:09.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.715+0000 7f5459587640 10 monclient: _check_auth_tickets 2024-03-29T11:03:09.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.715+0000 7f5459587640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:09.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.715+0000 7f5459587640 1 -- 172.21.15.71:0/2738510123 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f5444001670 con 0x7f5454142a40 2024-03-29T11:03:09.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.715+0000 7f5459587640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:09.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.715+0000 7f5459587640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:03:09.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.715+0000 7f5459587640 10 monclient: dump: 2024-03-29T11:03:09.702 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:09.702 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:09.702 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:09.702 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:09.703 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:09.703 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:09.703 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:09.703 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:09.703 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:09.703 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:09.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.715+0000 7f545c7ed640 5 monclient: authenticate success, global_id 4328 2024-03-29T11:03:09.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.715+0000 7f545c7ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:09.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.716+0000 7f5459587640 1 -- 172.21.15.71:0/2738510123 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54440018d0 con 0x7f5454142a40 2024-03-29T11:03:09.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.716+0000 7f5459587640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:09.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.716+0000 7f545c7ed640 10 monclient: get_monmap_and_config success 2024-03-29T11:03:09.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.716+0000 7f545c7ed640 4 set_mon_vals no callback set 2024-03-29T11:03:09.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.716+0000 7f5459587640 1 -- 172.21.15.71:0/2738510123 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f5444001670 con 0x7f5454142a40 2024-03-29T11:03:09.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.716+0000 7f5459587640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:09.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.716+0000 7f5459587640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:09.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.716+0000 7f5459587640 10 monclient: dump: 2024-03-29T11:03:09.704 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:09.704 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:09.704 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:09.704 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:09.704 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:09.704 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:09.704 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:09.704 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:09.704 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:09.704 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:09.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.717+0000 7f545c7ed640 10 monclient: shutdown 2024-03-29T11:03:09.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.717+0000 7f545c7ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:09.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.717+0000 7f545c7ed640 1 -- 172.21.15.71:0/2738510123 >> v1:172.21.15.71:6790/0 conn(0x7f5454142a40 legacy=0x7f545414d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:09.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.717+0000 7f545c7ed640 1 -- 172.21.15.71:0/2738510123 shutdown_connections 2024-03-29T11:03:09.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.717+0000 7f545c7ed640 1 -- 172.21.15.71:0/2738510123 >> 172.21.15.71:0/2738510123 conn(0x7f545406d0f0 msgr2=0x7f5454066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:09.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.717+0000 7f545c7ed640 1 -- 172.21.15.71:0/2738510123 shutdown_connections 2024-03-29T11:03:09.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.717+0000 7f545c7ed640 1 -- 172.21.15.71:0/2738510123 wait complete. 2024-03-29T11:03:09.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.718+0000 7f545c7ed640 10 monclient: build_initial_monmap 2024-03-29T11:03:09.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.718+0000 7f545c7ed640 10 monclient: monmap: 2024-03-29T11:03:09.708 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:03:09.708 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:03:09.708 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:03:09.708 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:03:09.708 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:09.708 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:09.708 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:03:09.708 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:03:09.708 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:03:09.708 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:09.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.719+0000 7f545c7ed640 1 librados: starting msgr at 2024-03-29T11:03:09.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.719+0000 7f545c7ed640 1 librados: starting objecter 2024-03-29T11:03:09.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.719+0000 7f545c7ed640 1 Processor -- start 2024-03-29T11:03:09.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.719+0000 7f545c7ed640 1 -- start start 2024-03-29T11:03:09.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.719+0000 7f545c7ed640 1 librados: setting wanted keys 2024-03-29T11:03:09.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.719+0000 7f545c7ed640 1 librados: calling monclient init 2024-03-29T11:03:09.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.719+0000 7f545c7ed640 10 monclient: init 2024-03-29T11:03:09.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.720+0000 7f545c7ed640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:09.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.720+0000 7f545c7ed640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:03:09.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.720+0000 7f545c7ed640 10 monclient(hunting): picked mon.noname-c con 0x7f54540b3da0 addr v1:172.21.15.71:6790/0 2024-03-29T11:03:09.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.720+0000 7f545c7ed640 10 monclient(hunting): picked mon.noname-b con 0x7f54540b46b0 addr v1:172.21.15.137:6789/0 2024-03-29T11:03:09.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.720+0000 7f545c7ed640 10 monclient(hunting): picked mon.noname-a con 0x7f5454142a40 addr v1:172.21.15.71:6789/0 2024-03-29T11:03:09.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.720+0000 7f545c7ed640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5454131a00 con 0x7f5454142a40 2024-03-29T11:03:09.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.720+0000 7f545c7ed640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54541d6d30 con 0x7f54540b46b0 2024-03-29T11:03:09.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.720+0000 7f545c7ed640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54541354b0 con 0x7f54540b3da0 2024-03-29T11:03:09.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.720+0000 7f545c7ed640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:09.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.720+0000 7f545c7ed640 10 monclient(hunting): authenticate will time out at 945.978592s 2024-03-29T11:03:09.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.727+0000 7f5459d88640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f54540b46b0 0x7f5454130d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:43931/0 (socket says 172.21.15.71:43931) 2024-03-29T11:03:09.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.727+0000 7f5459d88640 1 -- 172.21.15.71:0/2770561299 learned_addr learned my addr 172.21.15.71:0/2770561299 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:03:09.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.728+0000 7f544affd640 10 client.?.objecter ms_handle_connect 0x7f54540b3da0 2024-03-29T11:03:09.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.728+0000 7f544affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:09.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.728+0000 7f544affd640 1 -- 172.21.15.71:0/2770561299 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146508100 0 0) 0x7f54541354b0 con 0x7f54540b3da0 2024-03-29T11:03:09.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.728+0000 7f544affd640 10 monclient(hunting): _negotiate 2024-03-29T11:03:09.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.728+0000 7f544affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:09.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.728+0000 7f544affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:09.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.728+0000 7f544affd640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:03:09.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.728+0000 7f544affd640 1 -- 172.21.15.71:0/2770561299 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5438004eb0 con 0x7f54540b3da0 2024-03-29T11:03:09.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.728+0000 7f544affd640 1 -- 172.21.15.71:0/2770561299 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4003323223 0 0) 0x7f5438004eb0 con 0x7f54540b3da0 2024-03-29T11:03:09.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.728+0000 7f544affd640 1 -- 172.21.15.71:0/2770561299 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54541354b0 con 0x7f54540b3da0 2024-03-29T11:03:09.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.728+0000 7f544affd640 1 -- 172.21.15.71:0/2770561299 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f5450001870 con 0x7f54540b3da0 2024-03-29T11:03:09.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.728+0000 7f544affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:09.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.728+0000 7f544affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:03:09.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.728+0000 7f544affd640 10 monclient(hunting): dump: 2024-03-29T11:03:09.712 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:09.712 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:09.713 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:09.713 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:09.713 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:09.713 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:09.713 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:09.713 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:09.713 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:09.713 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:09.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.728+0000 7f544affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:03:09.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f544affd640 1 -- 172.21.15.71:0/2770561299 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2746776004 0 0) 0x7f54541354b0 con 0x7f54540b3da0 2024-03-29T11:03:09.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f544affd640 1 -- 172.21.15.71:0/2770561299 >> v1:172.21.15.137:6789/0 conn(0x7f54540b46b0 legacy=0x7f5454130d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:09.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f544affd640 1 -- 172.21.15.71:0/2770561299 >> v1:172.21.15.71:6789/0 conn(0x7f5454142a40 legacy=0x7f5454140ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:09.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f544affd640 10 monclient: _finish_hunting 0 2024-03-29T11:03:09.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f544affd640 1 monclient: found mon.c 2024-03-29T11:03:09.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f544affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:09.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f544affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:03:09.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f544affd640 1 -- 172.21.15.71:0/2770561299 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5454140050 con 0x7f54540b3da0 2024-03-29T11:03:09.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f544affd640 10 monclient: _finish_auth 0 2024-03-29T11:03:09.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f544affd640 10 monclient: _check_auth_tickets 2024-03-29T11:03:09.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f544affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:09.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f544affd640 10 client.?.objecter ms_handle_connect 0x7f5454142a40 2024-03-29T11:03:09.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f544affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:09.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f545c7ed640 5 monclient: authenticate success, global_id 4331 2024-03-29T11:03:09.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f545c7ed640 10 monclient: _renew_subs 2024-03-29T11:03:09.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f545c7ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:03:09.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f545c7ed640 1 -- 172.21.15.71:0/2770561299 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54541401e0 con 0x7f54540b3da0 2024-03-29T11:03:09.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f545c7ed640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:03:09.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f545c7ed640 10 monclient: _renew_subs 2024-03-29T11:03:09.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f545c7ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:03:09.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f545c7ed640 1 -- 172.21.15.71:0/2770561299 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5454131650 con 0x7f54540b3da0 2024-03-29T11:03:09.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f545c7ed640 1 librados: init done 2024-03-29T11:03:09.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f544affd640 1 -- 172.21.15.71:0/2770561299 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54500019d0 con 0x7f54540b3da0 2024-03-29T11:03:09.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f544affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:09.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f544affd640 1 -- 172.21.15.71:0/2770561299 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f5450003870 con 0x7f54540b3da0 2024-03-29T11:03:09.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f544affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:09.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f544affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:09.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.729+0000 7f544affd640 10 monclient: dump: 2024-03-29T11:03:09.715 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:09.715 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:09.716 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:09.716 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:09.716 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:09.716 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:09.716 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:09.716 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:09.716 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:09.716 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:09.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.730+0000 7f5458d86640 4 set_mon_vals no callback set 2024-03-29T11:03:09.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.731+0000 7f545c7ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:09.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.731+0000 7f545c7ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:09.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.731+0000 7f545c7ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:03:09.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.731+0000 7f545c7ed640 1 -- 172.21.15.71:0/2770561299 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f543c005080 con 0x7f54540b3da0 2024-03-29T11:03:09.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.734+0000 7f544affd640 1 -- 172.21.15.71:0/2770561299 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f54500039d0 con 0x7f54540b3da0 2024-03-29T11:03:09.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.735+0000 7f544affd640 1 -- 172.21.15.71:0/2770561299 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f5450083870 con 0x7f54540b3da0 2024-03-29T11:03:09.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.735+0000 7f5459d88640 10 monclient: get_auth_request con 0x7f5438076460 auth_method 0 2024-03-29T11:03:09.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.735+0000 7f544affd640 10 client.4331.objecter ms_dispatch 0x7f5454000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:03:09.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.735+0000 7f544affd640 3 client.4331.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:03:09.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.735+0000 7f544affd640 3 client.4331.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:03:09.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.735+0000 7f544affd640 20 client.4331.objecter dump_active .. 0 homeless 2024-03-29T11:03:09.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.735+0000 7f544affd640 10 client.4331.objecter ms_handle_connect 0x7f5438076460 2024-03-29T11:03:09.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.740+0000 7f544affd640 1 -- 172.21.15.71:0/2770561299 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f545004ef80 con 0x7f54540b3da0 2024-03-29T11:03:09.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.740+0000 7f544affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:09.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:09.740+0000 7f544affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:03:10.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:10.068+0000 7f545c7ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:10.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:10.068+0000 7f545c7ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:10.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:10.068+0000 7f545c7ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:03:10.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:10.068+0000 7f545c7ed640 1 -- 172.21.15.71:0/2770561299 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f543c005330 con 0x7f54540b3da0 2024-03-29T11:03:10.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:10.069+0000 7f544affd640 1 -- 172.21.15.71:0/2770561299 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f5450051080 con 0x7f54540b3da0 2024-03-29T11:03:10.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:10.069+0000 7f544affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:10.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:10.069+0000 7f544affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:03:10.051 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:03:10.051 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:03:10.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:10.075+0000 7f545c7ed640 10 librados: watch_flush enter 2024-03-29T11:03:10.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:10.075+0000 7f545c7ed640 10 librados: watch_flush exit 2024-03-29T11:03:10.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:10.075+0000 7f545c7ed640 20 client.4331.objecter shutdown clearing up homeless session... 2024-03-29T11:03:10.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:10.075+0000 7f545c7ed640 10 client.4331.objecter successfully canceled tick 2024-03-29T11:03:10.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:10.075+0000 7f545c7ed640 1 -- 172.21.15.71:0/2770561299 >> v1:172.21.15.137:6816/973332665 conn(0x7f5438076460 legacy=0x7f5438078920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:10.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:10.075+0000 7f545c7ed640 10 monclient: shutdown 2024-03-29T11:03:10.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:10.075+0000 7f545c7ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:10.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:10.075+0000 7f545c7ed640 1 -- 172.21.15.71:0/2770561299 >> v1:172.21.15.71:6790/0 conn(0x7f54540b3da0 legacy=0x7f54541d71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:10.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:10.075+0000 7f545c7ed640 1 -- 172.21.15.71:0/2770561299 shutdown_connections 2024-03-29T11:03:10.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:10.075+0000 7f545c7ed640 1 -- 172.21.15.71:0/2770561299 >> 172.21.15.71:0/2770561299 conn(0x7f545406d0f0 msgr2=0x7f54540b8270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:10.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:10.077+0000 7f545c7ed640 1 -- 172.21.15.71:0/2770561299 shutdown_connections 2024-03-29T11:03:10.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:10.077+0000 7f545c7ed640 1 -- 172.21.15.71:0/2770561299 wait complete. 2024-03-29T11:03:10.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:10.077+0000 7f545c7ed640 1 librados: shutdown 2024-03-29T11:03:10.075 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:03:16.882 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:03:17.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.060+0000 7f84d60f4640 10 monclient: get_monmap_and_config 2024-03-29T11:03:17.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.060+0000 7f84d60f4640 10 monclient: build_initial_monmap 2024-03-29T11:03:17.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.060+0000 7f84d60f4640 10 monclient: monmap: 2024-03-29T11:03:17.044 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:03:17.044 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:17.044 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:03:17.061785+0000 2024-03-29T11:03:17.044 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:03:17.061785+0000 2024-03-29T11:03:17.044 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:17.044 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:17.044 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:03:17.044 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:03:17.044 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:03:17.044 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:17.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.062+0000 7f84d60f4640 1 Processor -- start 2024-03-29T11:03:17.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.062+0000 7f84d60f4640 1 -- start start 2024-03-29T11:03:17.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.062+0000 7f84d60f4640 10 monclient: init 2024-03-29T11:03:17.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.063+0000 7f84d60f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:17.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.063+0000 7f84d60f4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:03:17.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.063+0000 7f84d60f4640 10 monclient(hunting): picked mon.noname-c con 0x7f84d00ab360 addr v1:172.21.15.137:6789/0 2024-03-29T11:03:17.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.063+0000 7f84d60f4640 10 monclient(hunting): picked mon.noname-a con 0x7f84d014fb00 addr v1:172.21.15.71:6789/0 2024-03-29T11:03:17.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.063+0000 7f84d60f4640 10 monclient(hunting): picked mon.noname-b con 0x7f84d0153090 addr v1:172.21.15.71:6790/0 2024-03-29T11:03:17.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.063+0000 7f84d60f4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d00a94e0 con 0x7f84d014fb00 2024-03-29T11:03:17.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.063+0000 7f84d60f4640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d00afb60 con 0x7f84d00ab360 2024-03-29T11:03:17.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.063+0000 7f84d60f4640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d00aab80 con 0x7f84d0153090 2024-03-29T11:03:17.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.063+0000 7f84d60f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:17.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.063+0000 7f84d60f4640 10 monclient(hunting): authenticate will time out at 953.321348s 2024-03-29T11:03:17.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.066+0000 7f84cf7fe640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f84d00ab360 0x7f84d00ad450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:57409/0 (socket says 172.21.15.71:57409) 2024-03-29T11:03:17.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.066+0000 7f84cf7fe640 1 -- 172.21.15.71:0/1607414914 learned_addr learned my addr 172.21.15.71:0/1607414914 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:03:17.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.067+0000 7f84ce7fc640 1 -- 172.21.15.71:0/1607414914 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534405058 0 0) 0x7f84d00a94e0 con 0x7f84d014fb00 2024-03-29T11:03:17.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.067+0000 7f84ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:03:17.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.067+0000 7f84ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:17.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.067+0000 7f84ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:17.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.067+0000 7f84ce7fc640 10 monclient(hunting): my global_id is 4341 2024-03-29T11:03:17.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.067+0000 7f84ce7fc640 1 -- 172.21.15.71:0/1607414914 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84bc003520 con 0x7f84d014fb00 2024-03-29T11:03:17.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.067+0000 7f84ce7fc640 1 -- 172.21.15.71:0/1607414914 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034070512 0 0) 0x7f84d00aab80 con 0x7f84d0153090 2024-03-29T11:03:17.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.067+0000 7f84ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:03:17.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.067+0000 7f84ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:17.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.067+0000 7f84ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:17.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.067+0000 7f84ce7fc640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:03:17.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.067+0000 7f84ce7fc640 1 -- 172.21.15.71:0/1607414914 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84d00a94e0 con 0x7f84d0153090 2024-03-29T11:03:17.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.068+0000 7f84ce7fc640 1 -- 172.21.15.71:0/1607414914 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1267244925 0 0) 0x7f84bc003520 con 0x7f84d014fb00 2024-03-29T11:03:17.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.068+0000 7f84ce7fc640 1 -- 172.21.15.71:0/1607414914 >> v1:172.21.15.71:6790/0 conn(0x7f84d0153090 legacy=0x7f84d015cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:17.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.068+0000 7f84ce7fc640 1 -- 172.21.15.71:0/1607414914 >> v1:172.21.15.137:6789/0 conn(0x7f84d00ab360 legacy=0x7f84d00ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:17.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.068+0000 7f84ce7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:03:17.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.068+0000 7f84ce7fc640 1 monclient: found mon.noname-a 2024-03-29T11:03:17.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.068+0000 7f84ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:17.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.068+0000 7f84ce7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:03:17.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.068+0000 7f84ce7fc640 1 -- 172.21.15.71:0/1607414914 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d01568d0 con 0x7f84d014fb00 2024-03-29T11:03:17.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.068+0000 7f84ce7fc640 10 monclient: _finish_auth 0 2024-03-29T11:03:17.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.068+0000 7f84ce7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:03:17.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.068+0000 7f84ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:17.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.068+0000 7f84ce7fc640 1 -- 172.21.15.71:0/1607414914 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f84b8001690 con 0x7f84d014fb00 2024-03-29T11:03:17.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.068+0000 7f84ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:17.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.068+0000 7f84ce7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:03:17.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.068+0000 7f84ce7fc640 10 monclient: dump: 2024-03-29T11:03:17.049 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:17.049 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:17.049 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:17.049 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:17.049 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:17.049 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:17.049 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:17.049 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:17.049 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:17.049 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:17.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.068+0000 7f84d60f4640 5 monclient: authenticate success, global_id 4341 2024-03-29T11:03:17.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.068+0000 7f84d60f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:17.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.068+0000 7f84ce7fc640 1 -- 172.21.15.71:0/1607414914 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84b80017f0 con 0x7f84d014fb00 2024-03-29T11:03:17.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.068+0000 7f84ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:17.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.068+0000 7f84ce7fc640 1 -- 172.21.15.71:0/1607414914 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f84b8001c00 con 0x7f84d014fb00 2024-03-29T11:03:17.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.068+0000 7f84ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:17.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.068+0000 7f84ce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:03:17.050 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.068+0000 7f84ce7fc640 10 monclient: dump: 2024-03-29T11:03:17.050 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:17.050 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:17.050 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:17.050 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:17.050 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:17.050 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:17.050 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:17.050 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:17.050 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:17.051 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:17.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.068+0000 7f84d60f4640 10 monclient: get_monmap_and_config success 2024-03-29T11:03:17.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.068+0000 7f84d60f4640 4 set_mon_vals no callback set 2024-03-29T11:03:17.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.069+0000 7f84d60f4640 10 monclient: shutdown 2024-03-29T11:03:17.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.069+0000 7f84d60f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:17.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.069+0000 7f84d60f4640 1 -- 172.21.15.71:0/1607414914 >> v1:172.21.15.71:6789/0 conn(0x7f84d014fb00 legacy=0x7f84d0152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:17.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.069+0000 7f84d60f4640 1 -- 172.21.15.71:0/1607414914 shutdown_connections 2024-03-29T11:03:17.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.069+0000 7f84d60f4640 1 -- 172.21.15.71:0/1607414914 >> 172.21.15.71:0/1607414914 conn(0x7f84d00b35e0 msgr2=0x7f84d00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:17.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.069+0000 7f84d60f4640 1 -- 172.21.15.71:0/1607414914 shutdown_connections 2024-03-29T11:03:17.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.069+0000 7f84d60f4640 1 -- 172.21.15.71:0/1607414914 wait complete. 2024-03-29T11:03:17.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.070+0000 7f84d60f4640 10 monclient: build_initial_monmap 2024-03-29T11:03:17.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.070+0000 7f84d60f4640 10 monclient: monmap: 2024-03-29T11:03:17.052 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:03:17.052 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:03:17.052 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:03:17.052 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:03:17.052 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:17.052 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:17.052 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:03:17.052 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:03:17.053 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:03:17.053 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:17.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.071+0000 7f84d60f4640 1 librados: starting msgr at 2024-03-29T11:03:17.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.071+0000 7f84d60f4640 1 librados: starting objecter 2024-03-29T11:03:17.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.071+0000 7f84d60f4640 1 Processor -- start 2024-03-29T11:03:17.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.071+0000 7f84d60f4640 1 -- start start 2024-03-29T11:03:17.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.071+0000 7f84d60f4640 1 librados: setting wanted keys 2024-03-29T11:03:17.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.071+0000 7f84d60f4640 1 librados: calling monclient init 2024-03-29T11:03:17.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.071+0000 7f84d60f4640 10 monclient: init 2024-03-29T11:03:17.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.072+0000 7f84d60f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:17.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.072+0000 7f84d60f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:03:17.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.072+0000 7f84d60f4640 10 monclient(hunting): picked mon.noname-c con 0x7f84d00ab360 addr v1:172.21.15.71:6790/0 2024-03-29T11:03:17.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.072+0000 7f84d60f4640 10 monclient(hunting): picked mon.noname-a con 0x7f84d014fb00 addr v1:172.21.15.71:6789/0 2024-03-29T11:03:17.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.072+0000 7f84d60f4640 10 monclient(hunting): picked mon.noname-b con 0x7f84d0153090 addr v1:172.21.15.137:6789/0 2024-03-29T11:03:17.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.072+0000 7f84d60f4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d006ee70 con 0x7f84d014fb00 2024-03-29T11:03:17.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.072+0000 7f84d60f4640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d0155110 con 0x7f84d0153090 2024-03-29T11:03:17.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.072+0000 7f84d60f4640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d01553d0 con 0x7f84d00ab360 2024-03-29T11:03:17.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.072+0000 7f84d60f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:17.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.072+0000 7f84d60f4640 10 monclient(hunting): authenticate will time out at 953.330786s 2024-03-29T11:03:17.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.074+0000 7f84cf7fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f84d00ab360 0x7f84d0153690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:33832/0 (socket says 172.21.15.71:33832) 2024-03-29T11:03:17.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.074+0000 7f84cf7fe640 1 -- 172.21.15.71:0/1513620300 learned_addr learned my addr 172.21.15.71:0/1513620300 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:03:17.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.074+0000 7f84c7fff640 10 client.?.objecter ms_handle_connect 0x7f84d00ab360 2024-03-29T11:03:17.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.074+0000 7f84c7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:17.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.074+0000 7f84c7fff640 1 -- 172.21.15.71:0/1513620300 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3682260281 0 0) 0x7f84d01553d0 con 0x7f84d00ab360 2024-03-29T11:03:17.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.074+0000 7f84c7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:03:17.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.074+0000 7f84c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:17.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.074+0000 7f84c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:17.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.074+0000 7f84c7fff640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:03:17.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.074+0000 7f84c7fff640 1 -- 172.21.15.71:0/1513620300 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84b0004eb0 con 0x7f84d00ab360 2024-03-29T11:03:17.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.075+0000 7f84c7fff640 1 -- 172.21.15.71:0/1513620300 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 941665329 0 0) 0x7f84b0004eb0 con 0x7f84d00ab360 2024-03-29T11:03:17.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.075+0000 7f84c7fff640 1 -- 172.21.15.71:0/1513620300 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84d01553d0 con 0x7f84d00ab360 2024-03-29T11:03:17.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.075+0000 7f84c7fff640 1 -- 172.21.15.71:0/1513620300 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f84c0001c30 con 0x7f84d00ab360 2024-03-29T11:03:17.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.075+0000 7f84c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:17.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.075+0000 7f84c7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:03:17.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.075+0000 7f84c7fff640 10 monclient(hunting): dump: 2024-03-29T11:03:17.058 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:17.058 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:17.058 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:17.058 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:17.058 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:17.058 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:17.058 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:17.058 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:17.058 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:17.058 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:17.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.075+0000 7f84c7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:03:17.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.075+0000 7f84c7fff640 1 -- 172.21.15.71:0/1513620300 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 344254459 0 0) 0x7f84d01553d0 con 0x7f84d00ab360 2024-03-29T11:03:17.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.075+0000 7f84c7fff640 1 -- 172.21.15.71:0/1513620300 >> v1:172.21.15.137:6789/0 conn(0x7f84d0153090 legacy=0x7f84d0150df0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:03:17.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.076+0000 7f84c7fff640 1 -- 172.21.15.71:0/1513620300 >> v1:172.21.15.71:6789/0 conn(0x7f84d014fb00 legacy=0x7f84d01558f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:03:17.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.076+0000 7f84c7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:03:17.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.076+0000 7f84c7fff640 1 monclient: found mon.c 2024-03-29T11:03:17.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.076+0000 7f84c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:17.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.076+0000 7f84c7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:03:17.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.076+0000 7f84c7fff640 1 -- 172.21.15.71:0/1513620300 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d01548a0 con 0x7f84d00ab360 2024-03-29T11:03:17.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.076+0000 7f84c7fff640 10 monclient: _finish_auth 0 2024-03-29T11:03:17.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.076+0000 7f84c7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:03:17.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.076+0000 7f84c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:17.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.076+0000 7f84d60f4640 5 monclient: authenticate success, global_id 4337 2024-03-29T11:03:17.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.076+0000 7f84d60f4640 10 monclient: _renew_subs 2024-03-29T11:03:17.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.076+0000 7f84d60f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:03:17.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.076+0000 7f84d60f4640 1 -- 172.21.15.71:0/1513620300 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d014d220 con 0x7f84d00ab360 2024-03-29T11:03:17.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.077+0000 7f84d60f4640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:03:17.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.077+0000 7f84d60f4640 10 monclient: _renew_subs 2024-03-29T11:03:17.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.077+0000 7f84d60f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:03:17.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.077+0000 7f84d60f4640 1 -- 172.21.15.71:0/1513620300 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d0163910 con 0x7f84d00ab360 2024-03-29T11:03:17.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.077+0000 7f84d60f4640 1 librados: init done 2024-03-29T11:03:17.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.078+0000 7f84c7fff640 1 -- 172.21.15.71:0/1513620300 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84c0001e90 con 0x7f84d00ab360 2024-03-29T11:03:17.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.078+0000 7f84c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:17.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.078+0000 7f84c7fff640 1 -- 172.21.15.71:0/1513620300 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f84c0004fb0 con 0x7f84d00ab360 2024-03-29T11:03:17.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.078+0000 7f84c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:17.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.078+0000 7f84c7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:17.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.078+0000 7f84c7fff640 10 monclient: dump: 2024-03-29T11:03:17.060 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:17.060 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:17.061 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:17.061 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:17.061 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:17.061 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:17.061 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:17.061 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:17.061 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:17.061 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:17.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.078+0000 7f84d60f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:17.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.078+0000 7f84d60f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:17.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.078+0000 7f84d60f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:03:17.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.078+0000 7f84d60f4640 1 -- 172.21.15.71:0/1513620300 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d00b3d60 con 0x7f84d00ab360 2024-03-29T11:03:17.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.078+0000 7f84cdffb640 4 set_mon_vals no callback set 2024-03-29T11:03:17.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.079+0000 7f84c7fff640 1 -- 172.21.15.71:0/1513620300 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f84c00051d0 con 0x7f84d00ab360 2024-03-29T11:03:17.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.079+0000 7f84c7fff640 1 -- 172.21.15.71:0/1513620300 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f84c0065db0 con 0x7f84d00ab360 2024-03-29T11:03:17.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.079+0000 7f84c7fff640 10 client.4337.objecter ms_dispatch 0x7f84d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:03:17.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.079+0000 7f84c7fff640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:03:17.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.079+0000 7f84c7fff640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:03:17.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.079+0000 7f84c7fff640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-29T11:03:17.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.080+0000 7f84ceffd640 10 monclient: get_auth_request con 0x7f84b006db90 auth_method 0 2024-03-29T11:03:17.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.080+0000 7f84c7fff640 10 client.4337.objecter ms_handle_connect 0x7f84b006db90 2024-03-29T11:03:17.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.087+0000 7f84c7fff640 1 -- 172.21.15.71:0/1513620300 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f84c0061730 con 0x7f84d00ab360 2024-03-29T11:03:17.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.087+0000 7f84c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:17.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.087+0000 7f84c7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:03:17.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.419+0000 7f84d60f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:17.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.419+0000 7f84d60f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:17.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.419+0000 7f84d60f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:03:17.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.419+0000 7f84d60f4640 1 -- 172.21.15.71:0/1513620300 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84d0154a30 con 0x7f84d00ab360 2024-03-29T11:03:17.400 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.420+0000 7f84c7fff640 1 -- 172.21.15.71:0/1513620300 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f84c00637f0 con 0x7f84d00ab360 2024-03-29T11:03:17.400 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.420+0000 7f84c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:17.400 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.420+0000 7f84c7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:03:17.401 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:03:17.401 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:03:17.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.425+0000 7f84d60f4640 10 librados: watch_flush enter 2024-03-29T11:03:17.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.425+0000 7f84d60f4640 10 librados: watch_flush exit 2024-03-29T11:03:17.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.425+0000 7f84d60f4640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-29T11:03:17.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.425+0000 7f84d60f4640 10 client.4337.objecter successfully canceled tick 2024-03-29T11:03:17.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.425+0000 7f84d60f4640 1 -- 172.21.15.71:0/1513620300 >> v1:172.21.15.137:6816/973332665 conn(0x7f84b006db90 legacy=0x7f84b0070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:17.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.425+0000 7f84d60f4640 10 monclient: shutdown 2024-03-29T11:03:17.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.425+0000 7f84d60f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:17.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.425+0000 7f84d60f4640 1 -- 172.21.15.71:0/1513620300 >> v1:172.21.15.71:6790/0 conn(0x7f84d00ab360 legacy=0x7f84d0153690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:17.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.425+0000 7f84d60f4640 1 -- 172.21.15.71:0/1513620300 shutdown_connections 2024-03-29T11:03:17.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.425+0000 7f84d60f4640 1 -- 172.21.15.71:0/1513620300 >> 172.21.15.71:0/1513620300 conn(0x7f84d00b35e0 msgr2=0x7f84d0162fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:17.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.426+0000 7f84d60f4640 1 -- 172.21.15.71:0/1513620300 shutdown_connections 2024-03-29T11:03:17.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.426+0000 7f84d60f4640 1 -- 172.21.15.71:0/1513620300 wait complete. 2024-03-29T11:03:17.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:17.426+0000 7f84d60f4640 1 librados: shutdown 2024-03-29T11:03:17.423 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:03:24.230 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:03:24.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.397+0000 7fb0df59e640 10 monclient: get_monmap_and_config 2024-03-29T11:03:24.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.397+0000 7fb0df59e640 10 monclient: build_initial_monmap 2024-03-29T11:03:24.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.397+0000 7fb0df59e640 10 monclient: monmap: 2024-03-29T11:03:24.377 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:03:24.377 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:24.378 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:03:24.398791+0000 2024-03-29T11:03:24.378 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:03:24.398791+0000 2024-03-29T11:03:24.378 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:24.378 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:24.378 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:03:24.378 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:03:24.378 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:03:24.378 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:24.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.398+0000 7fb0df59e640 1 Processor -- start 2024-03-29T11:03:24.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.398+0000 7fb0df59e640 1 -- start start 2024-03-29T11:03:24.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.398+0000 7fb0df59e640 10 monclient: init 2024-03-29T11:03:24.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.399+0000 7fb0df59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:24.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.399+0000 7fb0df59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:03:24.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.400+0000 7fb0df59e640 10 monclient(hunting): picked mon.noname-b con 0x7fb0d80389e0 addr v1:172.21.15.71:6790/0 2024-03-29T11:03:24.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.400+0000 7fb0df59e640 10 monclient(hunting): picked mon.noname-c con 0x7fb0d8038dc0 addr v1:172.21.15.137:6789/0 2024-03-29T11:03:24.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.400+0000 7fb0de59c640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fb0d80389e0 0x7fb0d8033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:55824/0 (socket says 172.21.15.71:55824) 2024-03-29T11:03:24.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.402+0000 7fb0df59e640 10 monclient(hunting): picked mon.noname-a con 0x7fb0d8030660 addr v1:172.21.15.71:6789/0 2024-03-29T11:03:24.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.403+0000 7fb0ded9d640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fb0d8030660 0x7fb0d80d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:39925/0 (socket says 172.21.15.71:39925) 2024-03-29T11:03:24.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.403+0000 7fb0de59c640 1 -- 172.21.15.71:0/3221185708 learned_addr learned my addr 172.21.15.71:0/3221185708 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:03:24.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.403+0000 7fb0df59e640 1 -- 172.21.15.71:0/3221185708 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d80c54c0 con 0x7fb0d8030660 2024-03-29T11:03:24.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.403+0000 7fb0df59e640 1 -- 172.21.15.71:0/3221185708 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d8036430 con 0x7fb0d8038dc0 2024-03-29T11:03:24.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.403+0000 7fb0df59e640 1 -- 172.21.15.71:0/3221185708 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d8039840 con 0x7fb0d80389e0 2024-03-29T11:03:24.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.403+0000 7fb0df59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:24.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.403+0000 7fb0df59e640 10 monclient(hunting): authenticate will time out at 960.661636s 2024-03-29T11:03:24.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.403+0000 7fb0dd59a640 1 -- 172.21.15.71:0/3221185708 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633883072 0 0) 0x7fb0d80c54c0 con 0x7fb0d8030660 2024-03-29T11:03:24.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.403+0000 7fb0dd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:03:24.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.403+0000 7fb0dd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:24.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.403+0000 7fb0dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:24.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.403+0000 7fb0dd59a640 10 monclient(hunting): my global_id is 4344 2024-03-29T11:03:24.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.404+0000 7fb0dd59a640 1 -- 172.21.15.71:0/3221185708 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0cc003520 con 0x7fb0d8030660 2024-03-29T11:03:24.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.405+0000 7fb0dd59a640 1 -- 172.21.15.71:0/3221185708 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2936084754 0 0) 0x7fb0cc003520 con 0x7fb0d8030660 2024-03-29T11:03:24.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.405+0000 7fb0dd59a640 1 -- 172.21.15.71:0/3221185708 >> v1:172.21.15.71:6790/0 conn(0x7fb0d80389e0 legacy=0x7fb0d8033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:24.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.405+0000 7fb0dd59a640 1 -- 172.21.15.71:0/3221185708 >> v1:172.21.15.137:6789/0 conn(0x7fb0d8038dc0 legacy=0x7fb0d80cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:03:24.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.407+0000 7fb0dd59a640 10 monclient: _finish_hunting 0 2024-03-29T11:03:24.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.407+0000 7fb0dd59a640 1 monclient: found mon.noname-a 2024-03-29T11:03:24.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.407+0000 7fb0dd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:24.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.407+0000 7fb0dd59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:03:24.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.407+0000 7fb0dd59a640 1 -- 172.21.15.71:0/3221185708 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d8035690 con 0x7fb0d8030660 2024-03-29T11:03:24.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.407+0000 7fb0dd59a640 10 monclient: _finish_auth 0 2024-03-29T11:03:24.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.407+0000 7fb0dd59a640 10 monclient: _check_auth_tickets 2024-03-29T11:03:24.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.407+0000 7fb0dd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:24.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.407+0000 7fb0df59e640 5 monclient: authenticate success, global_id 4344 2024-03-29T11:03:24.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.407+0000 7fb0df59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:24.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.407+0000 7fb0dd59a640 1 -- 172.21.15.71:0/3221185708 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb0e004f730 con 0x7fb0d8030660 2024-03-29T11:03:24.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.407+0000 7fb0dd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:24.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.407+0000 7fb0dd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:03:24.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.407+0000 7fb0dd59a640 10 monclient: dump: 2024-03-29T11:03:24.387 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:24.387 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:24.388 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:24.388 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:24.388 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:24.388 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:24.388 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:24.388 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:24.388 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:24.388 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:24.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.407+0000 7fb0df59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:24.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.408+0000 7fb0dd59a640 1 -- 172.21.15.71:0/3221185708 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0e0075f10 con 0x7fb0d8030660 2024-03-29T11:03:24.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.408+0000 7fb0dd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:24.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.408+0000 7fb0df59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:03:24.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.408+0000 7fb0df59e640 4 set_mon_vals no callback set 2024-03-29T11:03:24.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.408+0000 7fb0dd59a640 1 -- 172.21.15.71:0/3221185708 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb0e00758b0 con 0x7fb0d8030660 2024-03-29T11:03:24.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.408+0000 7fb0dd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:24.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.408+0000 7fb0dd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:03:24.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.408+0000 7fb0dd59a640 10 monclient: dump: 2024-03-29T11:03:24.390 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:24.390 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:24.390 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:24.390 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:24.390 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:24.390 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:24.390 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:24.390 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:24.390 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:24.390 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:24.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.409+0000 7fb0df59e640 10 monclient: shutdown 2024-03-29T11:03:24.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.409+0000 7fb0df59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:24.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.409+0000 7fb0df59e640 1 -- 172.21.15.71:0/3221185708 >> v1:172.21.15.71:6789/0 conn(0x7fb0d8030660 legacy=0x7fb0d80d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:24.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.412+0000 7fb0df59e640 1 -- 172.21.15.71:0/3221185708 shutdown_connections 2024-03-29T11:03:24.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.412+0000 7fb0df59e640 1 -- 172.21.15.71:0/3221185708 >> 172.21.15.71:0/3221185708 conn(0x7fb0d8005c00 msgr2=0x7fb0d8006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:24.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.412+0000 7fb0df59e640 1 -- 172.21.15.71:0/3221185708 shutdown_connections 2024-03-29T11:03:24.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.412+0000 7fb0df59e640 1 -- 172.21.15.71:0/3221185708 wait complete. 2024-03-29T11:03:24.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.413+0000 7fb0df59e640 10 monclient: build_initial_monmap 2024-03-29T11:03:24.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.413+0000 7fb0df59e640 10 monclient: monmap: 2024-03-29T11:03:24.393 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:03:24.393 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:03:24.393 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:03:24.394 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:03:24.394 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:24.394 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:24.394 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:03:24.394 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:03:24.394 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:03:24.394 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:24.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.413+0000 7fb0df59e640 1 librados: starting msgr at 2024-03-29T11:03:24.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.413+0000 7fb0df59e640 1 librados: starting objecter 2024-03-29T11:03:24.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.414+0000 7fb0df59e640 1 Processor -- start 2024-03-29T11:03:24.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.414+0000 7fb0df59e640 1 -- start start 2024-03-29T11:03:24.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.414+0000 7fb0df59e640 1 librados: setting wanted keys 2024-03-29T11:03:24.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.414+0000 7fb0df59e640 1 librados: calling monclient init 2024-03-29T11:03:24.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.414+0000 7fb0df59e640 10 monclient: init 2024-03-29T11:03:24.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.415+0000 7fb0df59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:24.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.415+0000 7fb0df59e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:03:24.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.416+0000 7fb0df59e640 10 monclient(hunting): picked mon.noname-a con 0x7fb0d8030660 addr v1:172.21.15.71:6789/0 2024-03-29T11:03:24.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.416+0000 7fb0df59e640 10 monclient(hunting): picked mon.noname-c con 0x7fb0d80389e0 addr v1:172.21.15.71:6790/0 2024-03-29T11:03:24.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.416+0000 7fb0ddd9b640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fb0d80389e0 0x7fb0d8169280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:57050/0 (socket says 172.21.15.71:57050) 2024-03-29T11:03:24.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.416+0000 7fb0ddd9b640 1 -- 172.21.15.71:0/402273085 learned_addr learned my addr 172.21.15.71:0/402273085 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:03:24.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.416+0000 7fb0df59e640 10 monclient(hunting): picked mon.noname-b con 0x7fb0d80c7a80 addr v1:172.21.15.137:6789/0 2024-03-29T11:03:24.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.416+0000 7fb0df59e640 1 -- 172.21.15.71:0/402273085 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d802fc40 con 0x7fb0d8030660 2024-03-29T11:03:24.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.416+0000 7fb0df59e640 1 -- 172.21.15.71:0/402273085 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d800a140 con 0x7fb0d80c7a80 2024-03-29T11:03:24.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.416+0000 7fb0df59e640 1 -- 172.21.15.71:0/402273085 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d802f8b0 con 0x7fb0d80389e0 2024-03-29T11:03:24.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.416+0000 7fb0df59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:24.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.416+0000 7fb0df59e640 10 monclient(hunting): authenticate will time out at 960.674534s 2024-03-29T11:03:24.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.416+0000 7fb0d6ffd640 10 client.?.objecter ms_handle_connect 0x7fb0d80389e0 2024-03-29T11:03:24.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.416+0000 7fb0d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:24.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.417+0000 7fb0d6ffd640 1 -- 172.21.15.71:0/402273085 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021189439 0 0) 0x7fb0d802f8b0 con 0x7fb0d80389e0 2024-03-29T11:03:24.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.417+0000 7fb0d6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:03:24.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.417+0000 7fb0d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:24.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.417+0000 7fb0d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:24.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.417+0000 7fb0d6ffd640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:03:24.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.417+0000 7fb0d6ffd640 1 -- 172.21.15.71:0/402273085 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0c0004eb0 con 0x7fb0d80389e0 2024-03-29T11:03:24.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.417+0000 7fb0d6ffd640 1 -- 172.21.15.71:0/402273085 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1863551185 0 0) 0x7fb0c0004eb0 con 0x7fb0d80389e0 2024-03-29T11:03:24.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.417+0000 7fb0d6ffd640 1 -- 172.21.15.71:0/402273085 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0d802f8b0 con 0x7fb0d80389e0 2024-03-29T11:03:24.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.417+0000 7fb0d6ffd640 1 -- 172.21.15.71:0/402273085 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb0c80017d0 con 0x7fb0d80389e0 2024-03-29T11:03:24.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.417+0000 7fb0d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:24.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.417+0000 7fb0d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:03:24.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.417+0000 7fb0d6ffd640 10 monclient(hunting): dump: 2024-03-29T11:03:24.398 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:24.398 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:24.398 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:24.398 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:24.398 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:24.398 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:24.398 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:24.398 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:24.398 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:24.398 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:24.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.417+0000 7fb0d6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:03:24.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.417+0000 7fb0d6ffd640 1 -- 172.21.15.71:0/402273085 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 815497385 0 0) 0x7fb0d802f8b0 con 0x7fb0d80389e0 2024-03-29T11:03:24.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.418+0000 7fb0d6ffd640 1 -- 172.21.15.71:0/402273085 >> v1:172.21.15.137:6789/0 conn(0x7fb0d80c7a80 legacy=0x7fb0d8169990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:03:24.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.418+0000 7fb0d6ffd640 1 -- 172.21.15.71:0/402273085 >> v1:172.21.15.71:6789/0 conn(0x7fb0d8030660 legacy=0x7fb0d802f1a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:03:24.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.420+0000 7fb0d6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:03:24.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.420+0000 7fb0d6ffd640 1 monclient: found mon.c 2024-03-29T11:03:24.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.420+0000 7fb0d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:24.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.420+0000 7fb0d6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:03:24.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.420+0000 7fb0d6ffd640 1 -- 172.21.15.71:0/402273085 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d8176da0 con 0x7fb0d80389e0 2024-03-29T11:03:24.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.420+0000 7fb0d6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:03:24.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.420+0000 7fb0d6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:03:24.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.420+0000 7fb0d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:24.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.420+0000 7fb0df59e640 5 monclient: authenticate success, global_id 4343 2024-03-29T11:03:24.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.420+0000 7fb0df59e640 10 monclient: _renew_subs 2024-03-29T11:03:24.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.420+0000 7fb0df59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:03:24.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.420+0000 7fb0df59e640 1 -- 172.21.15.71:0/402273085 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d8174d80 con 0x7fb0d80389e0 2024-03-29T11:03:24.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.421+0000 7fb0df59e640 10 client.4343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:03:24.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.421+0000 7fb0df59e640 10 monclient: _renew_subs 2024-03-29T11:03:24.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.421+0000 7fb0df59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:03:24.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.421+0000 7fb0df59e640 1 -- 172.21.15.71:0/402273085 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d8175300 con 0x7fb0d80389e0 2024-03-29T11:03:24.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.421+0000 7fb0df59e640 1 librados: init done 2024-03-29T11:03:24.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.421+0000 7fb0d6ffd640 1 -- 172.21.15.71:0/402273085 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0c8001a30 con 0x7fb0d80389e0 2024-03-29T11:03:24.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.421+0000 7fb0d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:24.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.421+0000 7fb0d6ffd640 1 -- 172.21.15.71:0/402273085 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb0c8004a40 con 0x7fb0d80389e0 2024-03-29T11:03:24.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.421+0000 7fb0d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:24.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.421+0000 7fb0d6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:24.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.421+0000 7fb0d6ffd640 10 monclient: dump: 2024-03-29T11:03:24.403 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:24.403 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:24.403 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:24.403 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:24.403 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:24.403 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:24.403 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:24.403 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:24.403 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:24.403 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:24.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.421+0000 7fb0d7fff640 4 set_mon_vals no callback set 2024-03-29T11:03:24.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.423+0000 7fb0d4ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:24.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.423+0000 7fb0d4ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:24.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.423+0000 7fb0d4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:03:24.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.423+0000 7fb0d4ff9640 1 -- 172.21.15.71:0/402273085 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0d8007b90 con 0x7fb0d80389e0 2024-03-29T11:03:24.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.424+0000 7fb0d6ffd640 1 -- 172.21.15.71:0/402273085 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7fb0c801aa10 con 0x7fb0d80389e0 2024-03-29T11:03:24.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.425+0000 7fb0de59c640 10 monclient: get_auth_request con 0x7fb0c006dc10 auth_method 0 2024-03-29T11:03:24.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.425+0000 7fb0d6ffd640 1 -- 172.21.15.71:0/402273085 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7fb0c8084b30 con 0x7fb0d80389e0 2024-03-29T11:03:24.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.425+0000 7fb0d6ffd640 10 client.4343.objecter ms_dispatch 0x7fb0e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:03:24.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.425+0000 7fb0d6ffd640 3 client.4343.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:03:24.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.425+0000 7fb0d6ffd640 3 client.4343.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:03:24.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.425+0000 7fb0d6ffd640 20 client.4343.objecter dump_active .. 0 homeless 2024-03-29T11:03:24.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.425+0000 7fb0d6ffd640 10 client.4343.objecter ms_handle_connect 0x7fb0c006dc10 2024-03-29T11:03:24.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.431+0000 7fb0d6ffd640 1 -- 172.21.15.71:0/402273085 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb0c8050240 con 0x7fb0d80389e0 2024-03-29T11:03:24.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.432+0000 7fb0d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:24.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.432+0000 7fb0d6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:03:24.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.742+0000 7fb0d4ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:24.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.742+0000 7fb0d4ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:24.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.742+0000 7fb0d4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:03:24.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.742+0000 7fb0d4ff9640 1 -- 172.21.15.71:0/402273085 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0d8174f10 con 0x7fb0d80389e0 2024-03-29T11:03:24.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.744+0000 7fb0d6ffd640 1 -- 172.21.15.71:0/402273085 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7fb0c8052340 con 0x7fb0d80389e0 2024-03-29T11:03:24.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.744+0000 7fb0d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:24.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.744+0000 7fb0d6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:03:24.724 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:03:24.724 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:03:24.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.747+0000 7fb0d4ff9640 10 librados: watch_flush enter 2024-03-29T11:03:24.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.748+0000 7fb0d4ff9640 10 librados: watch_flush exit 2024-03-29T11:03:24.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.748+0000 7fb0d4ff9640 20 client.4343.objecter shutdown clearing up homeless session... 2024-03-29T11:03:24.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.748+0000 7fb0d4ff9640 10 client.4343.objecter successfully canceled tick 2024-03-29T11:03:24.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.748+0000 7fb0d4ff9640 1 -- 172.21.15.71:0/402273085 >> v1:172.21.15.137:6816/973332665 conn(0x7fb0c006dc10 legacy=0x7fb0c00700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:24.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.748+0000 7fb0d4ff9640 10 monclient: shutdown 2024-03-29T11:03:24.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.748+0000 7fb0d4ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:24.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.748+0000 7fb0d4ff9640 1 -- 172.21.15.71:0/402273085 >> v1:172.21.15.71:6790/0 conn(0x7fb0d80389e0 legacy=0x7fb0d8169280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:24.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.748+0000 7fb0d4ff9640 1 -- 172.21.15.71:0/402273085 shutdown_connections 2024-03-29T11:03:24.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.748+0000 7fb0d4ff9640 1 -- 172.21.15.71:0/402273085 >> 172.21.15.71:0/402273085 conn(0x7fb0d8005c00 msgr2=0x7fb0d80c4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:24.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.749+0000 7fb0d4ff9640 1 -- 172.21.15.71:0/402273085 shutdown_connections 2024-03-29T11:03:24.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.749+0000 7fb0d4ff9640 1 -- 172.21.15.71:0/402273085 wait complete. 2024-03-29T11:03:24.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:24.750+0000 7fb0d4ff9640 1 librados: shutdown 2024-03-29T11:03:24.747 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:03:31.554 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:03:31.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.724+0000 7f877b3e6640 10 monclient: get_monmap_and_config 2024-03-29T11:03:31.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.724+0000 7f877b3e6640 10 monclient: build_initial_monmap 2024-03-29T11:03:31.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.724+0000 7f877b3e6640 10 monclient: monmap: 2024-03-29T11:03:31.705 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:03:31.705 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:31.705 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:03:31.725250+0000 2024-03-29T11:03:31.705 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:03:31.725250+0000 2024-03-29T11:03:31.705 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:31.705 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:31.705 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:03:31.706 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:03:31.706 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:03:31.706 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:31.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.725+0000 7f877b3e6640 1 Processor -- start 2024-03-29T11:03:31.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.726+0000 7f877b3e6640 1 -- start start 2024-03-29T11:03:31.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.726+0000 7f877b3e6640 10 monclient: init 2024-03-29T11:03:31.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.726+0000 7f877b3e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:31.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.727+0000 7f877b3e6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:03:31.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.727+0000 7f877b3e6640 10 monclient(hunting): picked mon.noname-c con 0x7f87740ae250 addr v1:172.21.15.137:6789/0 2024-03-29T11:03:31.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.727+0000 7f877b3e6640 10 monclient(hunting): picked mon.noname-b con 0x7f877413f970 addr v1:172.21.15.71:6790/0 2024-03-29T11:03:31.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.727+0000 7f877b3e6640 10 monclient(hunting): picked mon.noname-a con 0x7f8774142800 addr v1:172.21.15.71:6789/0 2024-03-29T11:03:31.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.729+0000 7f877b3e6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87740afe00 con 0x7f8774142800 2024-03-29T11:03:31.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.729+0000 7f877b3e6640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8774142c10 con 0x7f87740ae250 2024-03-29T11:03:31.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.729+0000 7f877b3e6640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87740afb20 con 0x7f877413f970 2024-03-29T11:03:31.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.729+0000 7f877b3e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:31.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.729+0000 7f877b3e6640 10 monclient(hunting): authenticate will time out at 967.987491s 2024-03-29T11:03:31.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.729+0000 7f8778981640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f877413f970 0x7f87741420f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:41208/0 (socket says 172.21.15.71:41208) 2024-03-29T11:03:31.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.729+0000 7f8778981640 1 -- 172.21.15.71:0/1088448802 learned_addr learned my addr 172.21.15.71:0/1088448802 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:03:31.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.730+0000 7f876bfff640 1 -- 172.21.15.71:0/1088448802 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4006260009 0 0) 0x7f8774142c10 con 0x7f87740ae250 2024-03-29T11:03:31.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.730+0000 7f876bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:03:31.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.730+0000 7f876bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:31.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.730+0000 7f876bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:31.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.730+0000 7f876bfff640 10 monclient(hunting): my global_id is 4306 2024-03-29T11:03:31.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.730+0000 7f876bfff640 1 -- 172.21.15.71:0/1088448802 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8760003520 con 0x7f87740ae250 2024-03-29T11:03:31.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.730+0000 7f876bfff640 1 -- 172.21.15.71:0/1088448802 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381113515 0 0) 0x7f87740afb20 con 0x7f877413f970 2024-03-29T11:03:31.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.730+0000 7f876bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:03:31.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.730+0000 7f876bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:31.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.730+0000 7f876bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:31.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.730+0000 7f876bfff640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:03:31.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.730+0000 7f876bfff640 1 -- 172.21.15.71:0/1088448802 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8774142c10 con 0x7f877413f970 2024-03-29T11:03:31.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.730+0000 7f876bfff640 1 -- 172.21.15.71:0/1088448802 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2095591938 0 0) 0x7f8774142c10 con 0x7f877413f970 2024-03-29T11:03:31.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.730+0000 7f876bfff640 1 -- 172.21.15.71:0/1088448802 >> v1:172.21.15.137:6789/0 conn(0x7f87740ae250 legacy=0x7f877413f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:31.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.730+0000 7f876bfff640 1 -- 172.21.15.71:0/1088448802 >> v1:172.21.15.71:6789/0 conn(0x7f8774142800 legacy=0x7f877414da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:31.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.730+0000 7f876bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:03:31.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.730+0000 7f876bfff640 1 monclient: found mon.noname-b 2024-03-29T11:03:31.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.730+0000 7f876bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:31.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.730+0000 7f876bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:03:31.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.730+0000 7f876bfff640 1 -- 172.21.15.71:0/1088448802 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87740aab80 con 0x7f877413f970 2024-03-29T11:03:31.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.730+0000 7f876bfff640 10 monclient: _finish_auth 0 2024-03-29T11:03:31.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.730+0000 7f876bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:03:31.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.730+0000 7f876bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:31.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.730+0000 7f876bfff640 1 -- 172.21.15.71:0/1088448802 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f8764001670 con 0x7f877413f970 2024-03-29T11:03:31.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.730+0000 7f877b3e6640 5 monclient: authenticate success, global_id 4346 2024-03-29T11:03:31.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.730+0000 7f877b3e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:31.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.733+0000 7f876bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:31.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.733+0000 7f876bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:03:31.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.733+0000 7f876bfff640 10 monclient: dump: 2024-03-29T11:03:31.715 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:31.716 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:31.716 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:31.716 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:31.716 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:31.716 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:31.716 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:31.716 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:31.716 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:31.716 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:31.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.733+0000 7f876bfff640 1 -- 172.21.15.71:0/1088448802 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8764001890 con 0x7f877413f970 2024-03-29T11:03:31.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.733+0000 7f876bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:31.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.733+0000 7f876bfff640 1 -- 172.21.15.71:0/1088448802 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f8764001c20 con 0x7f877413f970 2024-03-29T11:03:31.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.733+0000 7f876bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:31.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.733+0000 7f876bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:31.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.733+0000 7f876bfff640 10 monclient: dump: 2024-03-29T11:03:31.717 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:31.717 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:31.717 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:31.717 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:31.717 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:31.717 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:31.717 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:31.717 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:31.717 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:31.717 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:31.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.733+0000 7f877b3e6640 10 monclient: get_monmap_and_config success 2024-03-29T11:03:31.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.733+0000 7f877b3e6640 4 set_mon_vals no callback set 2024-03-29T11:03:31.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.734+0000 7f877b3e6640 10 monclient: shutdown 2024-03-29T11:03:31.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.734+0000 7f877b3e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:31.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.734+0000 7f877b3e6640 1 -- 172.21.15.71:0/1088448802 >> v1:172.21.15.71:6790/0 conn(0x7f877413f970 legacy=0x7f87741420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:31.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.737+0000 7f877b3e6640 1 -- 172.21.15.71:0/1088448802 shutdown_connections 2024-03-29T11:03:31.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.737+0000 7f877b3e6640 1 -- 172.21.15.71:0/1088448802 >> 172.21.15.71:0/1088448802 conn(0x7f87740b35e0 msgr2=0x7f87740b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:31.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.738+0000 7f877b3e6640 1 -- 172.21.15.71:0/1088448802 shutdown_connections 2024-03-29T11:03:31.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.738+0000 7f877b3e6640 1 -- 172.21.15.71:0/1088448802 wait complete. 2024-03-29T11:03:31.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.739+0000 7f877b3e6640 10 monclient: build_initial_monmap 2024-03-29T11:03:31.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.739+0000 7f877b3e6640 10 monclient: monmap: 2024-03-29T11:03:31.719 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:03:31.719 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:03:31.719 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:03:31.719 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:03:31.719 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:31.719 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:31.719 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:03:31.719 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:03:31.720 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:03:31.720 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:31.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.740+0000 7f877b3e6640 1 librados: starting msgr at 2024-03-29T11:03:31.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.740+0000 7f877b3e6640 1 librados: starting objecter 2024-03-29T11:03:31.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.740+0000 7f877b3e6640 1 Processor -- start 2024-03-29T11:03:31.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.740+0000 7f877b3e6640 1 -- start start 2024-03-29T11:03:31.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.740+0000 7f877b3e6640 1 librados: setting wanted keys 2024-03-29T11:03:31.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.740+0000 7f877b3e6640 1 librados: calling monclient init 2024-03-29T11:03:31.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.740+0000 7f877b3e6640 10 monclient: init 2024-03-29T11:03:31.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.741+0000 7f877b3e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:31.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.741+0000 7f877b3e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:03:31.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.741+0000 7f877b3e6640 10 monclient(hunting): picked mon.noname-c con 0x7f87740ae250 addr v1:172.21.15.71:6790/0 2024-03-29T11:03:31.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.741+0000 7f877b3e6640 10 monclient(hunting): picked mon.noname-a con 0x7f877413f970 addr v1:172.21.15.71:6789/0 2024-03-29T11:03:31.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.741+0000 7f877b3e6640 10 monclient(hunting): picked mon.noname-b con 0x7f8774142800 addr v1:172.21.15.137:6789/0 2024-03-29T11:03:31.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.742+0000 7f8779182640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f87740ae250 0x7f877413b5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:45282/0 (socket says 172.21.15.71:45282) 2024-03-29T11:03:31.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.743+0000 7f8779182640 1 -- 172.21.15.71:0/2468604214 learned_addr learned my addr 172.21.15.71:0/2468604214 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:03:31.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.743+0000 7f877b3e6640 1 -- 172.21.15.71:0/2468604214 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f877413beb0 con 0x7f877413f970 2024-03-29T11:03:31.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.743+0000 7f877b3e6640 1 -- 172.21.15.71:0/2468604214 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f877413c340 con 0x7f8774142800 2024-03-29T11:03:31.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.744+0000 7f877b3e6640 1 -- 172.21.15.71:0/2468604214 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8774143d80 con 0x7f87740ae250 2024-03-29T11:03:31.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.744+0000 7f877b3e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:31.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.744+0000 7f877b3e6640 10 monclient(hunting): authenticate will time out at 968.002271s 2024-03-29T11:03:31.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.745+0000 7f8769ffb640 10 client.?.objecter ms_handle_connect 0x7f87740ae250 2024-03-29T11:03:31.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.745+0000 7f8769ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:31.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.747+0000 7f8769ffb640 1 -- 172.21.15.71:0/2468604214 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1996405363 0 0) 0x7f8774143d80 con 0x7f87740ae250 2024-03-29T11:03:31.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.747+0000 7f8769ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:03:31.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.747+0000 7f8769ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:31.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.747+0000 7f8769ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:31.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.747+0000 7f8769ffb640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:03:31.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.747+0000 7f8769ffb640 1 -- 172.21.15.71:0/2468604214 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8758004eb0 con 0x7f87740ae250 2024-03-29T11:03:31.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.747+0000 7f8769ffb640 10 client.?.objecter ms_handle_connect 0x7f877413f970 2024-03-29T11:03:31.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.747+0000 7f8769ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:31.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.747+0000 7f8769ffb640 10 client.?.objecter ms_handle_connect 0x7f8774142800 2024-03-29T11:03:31.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.747+0000 7f8769ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:31.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.747+0000 7f8769ffb640 1 -- 172.21.15.71:0/2468604214 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 907572175 0 0) 0x7f8758004eb0 con 0x7f87740ae250 2024-03-29T11:03:31.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 1 -- 172.21.15.71:0/2468604214 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8774143d80 con 0x7f87740ae250 2024-03-29T11:03:31.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 1 -- 172.21.15.71:0/2468604214 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f87700019e0 con 0x7f87740ae250 2024-03-29T11:03:31.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:31.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:03:31.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 10 monclient(hunting): dump: 2024-03-29T11:03:31.730 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:31.730 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:31.730 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:31.731 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:31.731 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:31.731 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:31.731 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:31.731 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:31.731 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:31.731 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:31.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:03:31.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 1 -- 172.21.15.71:0/2468604214 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 341266392 0 0) 0x7f877413beb0 con 0x7f877413f970 2024-03-29T11:03:31.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:03:31.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:31.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:31.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 10 monclient(hunting): my global_id is 4350 2024-03-29T11:03:31.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 1 -- 172.21.15.71:0/2468604214 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8758004eb0 con 0x7f877413f970 2024-03-29T11:03:31.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 1 -- 172.21.15.71:0/2468604214 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3063034081 0 0) 0x7f8774143d80 con 0x7f87740ae250 2024-03-29T11:03:31.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 1 -- 172.21.15.71:0/2468604214 >> v1:172.21.15.137:6789/0 conn(0x7f8774142800 legacy=0x7f87741e1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:31.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 1 -- 172.21.15.71:0/2468604214 >> v1:172.21.15.71:6789/0 conn(0x7f877413f970 legacy=0x7f87741d6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:31.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:03:31.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 1 monclient: found mon.c 2024-03-29T11:03:31.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:31.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:03:31.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 1 -- 172.21.15.71:0/2468604214 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8774143ff0 con 0x7f87740ae250 2024-03-29T11:03:31.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 10 monclient: _finish_auth 0 2024-03-29T11:03:31.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:03:31.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:31.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 1 -- 172.21.15.71:0/2468604214 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8770001b40 con 0x7f87740ae250 2024-03-29T11:03:31.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:31.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 1 -- 172.21.15.71:0/2468604214 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f8770003a40 con 0x7f87740ae250 2024-03-29T11:03:31.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:31.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:31.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f8769ffb640 10 monclient: dump: 2024-03-29T11:03:31.735 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:31.735 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:31.735 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:31.735 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:31.735 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:31.735 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:31.736 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:31.736 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:31.736 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:31.736 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:31.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f877b3e6640 5 monclient: authenticate success, global_id 4349 2024-03-29T11:03:31.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f877b3e6640 10 monclient: _renew_subs 2024-03-29T11:03:31.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f877b3e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:03:31.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f877b3e6640 1 -- 172.21.15.71:0/2468604214 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87740ac740 con 0x7f87740ae250 2024-03-29T11:03:31.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.748+0000 7f876b7fe640 4 set_mon_vals no callback set 2024-03-29T11:03:31.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.749+0000 7f877b3e6640 10 client.4349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:03:31.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.749+0000 7f877b3e6640 10 monclient: _renew_subs 2024-03-29T11:03:31.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.749+0000 7f877b3e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:03:31.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.749+0000 7f877b3e6640 1 -- 172.21.15.71:0/2468604214 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87741d7830 con 0x7f87740ae250 2024-03-29T11:03:31.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.749+0000 7f877b3e6640 1 librados: init done 2024-03-29T11:03:31.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.750+0000 7f8769ffb640 1 -- 172.21.15.71:0/2468604214 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f8770003ba0 con 0x7f87740ae250 2024-03-29T11:03:31.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.751+0000 7f8769ffb640 1 -- 172.21.15.71:0/2468604214 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f8770084ac0 con 0x7f87740ae250 2024-03-29T11:03:31.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.751+0000 7f8769ffb640 10 client.4349.objecter ms_dispatch 0x7f8774000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:03:31.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.751+0000 7f8769ffb640 3 client.4349.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:03:31.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.751+0000 7f8769ffb640 3 client.4349.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:03:31.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.751+0000 7f8769ffb640 20 client.4349.objecter dump_active .. 0 homeless 2024-03-29T11:03:31.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.752+0000 7f8778981640 10 monclient: get_auth_request con 0x7f875806dd60 auth_method 0 2024-03-29T11:03:31.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.752+0000 7f8769ffb640 10 client.4349.objecter ms_handle_connect 0x7f875806dd60 2024-03-29T11:03:31.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.753+0000 7f877b3e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:31.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.753+0000 7f877b3e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:31.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.753+0000 7f877b3e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:03:31.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.753+0000 7f877b3e6640 1 -- 172.21.15.71:0/2468604214 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8774075f10 con 0x7f87740ae250 2024-03-29T11:03:31.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.761+0000 7f8769ffb640 1 -- 172.21.15.71:0/2468604214 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f87700503a0 con 0x7f87740ae250 2024-03-29T11:03:31.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.762+0000 7f8769ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:31.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:31.762+0000 7f8769ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:03:32.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:32.084+0000 7f877b3e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:32.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:32.084+0000 7f877b3e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:32.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:32.084+0000 7f877b3e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:03:32.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:32.084+0000 7f877b3e6640 1 -- 172.21.15.71:0/2468604214 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f877406ab70 con 0x7f87740ae250 2024-03-29T11:03:32.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:32.087+0000 7f8769ffb640 1 -- 172.21.15.71:0/2468604214 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f87700522d0 con 0x7f87740ae250 2024-03-29T11:03:32.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:32.087+0000 7f8769ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:32.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:32.087+0000 7f8769ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:03:32.067 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:03:32.067 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:03:32.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:32.090+0000 7f877b3e6640 10 librados: watch_flush enter 2024-03-29T11:03:32.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:32.090+0000 7f877b3e6640 10 librados: watch_flush exit 2024-03-29T11:03:32.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:32.091+0000 7f877b3e6640 20 client.4349.objecter shutdown clearing up homeless session... 2024-03-29T11:03:32.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:32.091+0000 7f877b3e6640 10 client.4349.objecter successfully canceled tick 2024-03-29T11:03:32.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:32.091+0000 7f877b3e6640 1 -- 172.21.15.71:0/2468604214 >> v1:172.21.15.137:6816/973332665 conn(0x7f875806dd60 legacy=0x7f8758070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:32.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:32.091+0000 7f877b3e6640 10 monclient: shutdown 2024-03-29T11:03:32.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:32.091+0000 7f877b3e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:32.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:32.091+0000 7f877b3e6640 1 -- 172.21.15.71:0/2468604214 >> v1:172.21.15.71:6790/0 conn(0x7f87740ae250 legacy=0x7f877413b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:32.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:32.091+0000 7f877b3e6640 1 -- 172.21.15.71:0/2468604214 shutdown_connections 2024-03-29T11:03:32.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:32.091+0000 7f877b3e6640 1 -- 172.21.15.71:0/2468604214 >> 172.21.15.71:0/2468604214 conn(0x7f87740b35e0 msgr2=0x7f87740b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:32.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:32.091+0000 7f877b3e6640 1 -- 172.21.15.71:0/2468604214 shutdown_connections 2024-03-29T11:03:32.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:32.092+0000 7f877b3e6640 1 -- 172.21.15.71:0/2468604214 wait complete. 2024-03-29T11:03:32.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:32.092+0000 7f877b3e6640 1 librados: shutdown 2024-03-29T11:03:32.088 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:03:38.895 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:03:39.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.063+0000 7f9131ecf640 10 monclient: get_monmap_and_config 2024-03-29T11:03:39.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.063+0000 7f9131ecf640 10 monclient: build_initial_monmap 2024-03-29T11:03:39.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.063+0000 7f9131ecf640 10 monclient: monmap: 2024-03-29T11:03:39.044 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:03:39.044 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:39.044 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:03:39.065404+0000 2024-03-29T11:03:39.044 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:03:39.065404+0000 2024-03-29T11:03:39.044 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:39.044 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:39.044 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:03:39.044 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:03:39.044 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:03:39.044 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:39.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.068+0000 7f9131ecf640 1 Processor -- start 2024-03-29T11:03:39.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.068+0000 7f9131ecf640 1 -- start start 2024-03-29T11:03:39.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.068+0000 7f9131ecf640 10 monclient: init 2024-03-29T11:03:39.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.069+0000 7f9131ecf640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:39.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.069+0000 7f9131ecf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:03:39.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.069+0000 7f9131ecf640 10 monclient(hunting): picked mon.noname-c con 0x7f912c1422b0 addr v1:172.21.15.137:6789/0 2024-03-29T11:03:39.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.069+0000 7f9131ecf640 10 monclient(hunting): picked mon.noname-a con 0x7f912c142690 addr v1:172.21.15.71:6789/0 2024-03-29T11:03:39.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.069+0000 7f9131ecf640 10 monclient(hunting): picked mon.noname-b con 0x7f912c0b17d0 addr v1:172.21.15.71:6790/0 2024-03-29T11:03:39.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.069+0000 7f9131ecf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f912c142c80 con 0x7f912c142690 2024-03-29T11:03:39.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.069+0000 7f9131ecf640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f912c0aae40 con 0x7f912c1422b0 2024-03-29T11:03:39.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.069+0000 7f9131ecf640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f912c13fbd0 con 0x7f912c0b17d0 2024-03-29T11:03:39.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.069+0000 7f9131ecf640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:39.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.069+0000 7f9131ecf640 10 monclient(hunting): authenticate will time out at 975.328312s 2024-03-29T11:03:39.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.072+0000 7f912bfff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f912c0b17d0 0x7f912c0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:54611/0 (socket says 172.21.15.71:54611) 2024-03-29T11:03:39.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.072+0000 7f912bfff640 1 -- 172.21.15.71:0/4294566903 learned_addr learned my addr 172.21.15.71:0/4294566903 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:03:39.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.073+0000 7f912a7fc640 1 -- 172.21.15.71:0/4294566903 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203001980 0 0) 0x7f912c142c80 con 0x7f912c142690 2024-03-29T11:03:39.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.073+0000 7f912a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:03:39.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.073+0000 7f912a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:39.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.073+0000 7f912a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:39.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.073+0000 7f912a7fc640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:03:39.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.073+0000 7f912a7fc640 1 -- 172.21.15.71:0/4294566903 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9114003520 con 0x7f912c142690 2024-03-29T11:03:39.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.073+0000 7f912a7fc640 1 -- 172.21.15.71:0/4294566903 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 105903593 0 0) 0x7f912c13fbd0 con 0x7f912c0b17d0 2024-03-29T11:03:39.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.073+0000 7f912a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:03:39.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.073+0000 7f912a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:39.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.073+0000 7f912a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:39.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.073+0000 7f912a7fc640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:03:39.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.073+0000 7f912a7fc640 1 -- 172.21.15.71:0/4294566903 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91140048a0 con 0x7f912c0b17d0 2024-03-29T11:03:39.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.074+0000 7f912a7fc640 1 -- 172.21.15.71:0/4294566903 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 124485859 0 0) 0x7f91140048a0 con 0x7f912c0b17d0 2024-03-29T11:03:39.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.074+0000 7f912a7fc640 1 -- 172.21.15.71:0/4294566903 >> v1:172.21.15.137:6789/0 conn(0x7f912c1422b0 legacy=0x7f912c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:39.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.074+0000 7f912a7fc640 1 -- 172.21.15.71:0/4294566903 >> v1:172.21.15.71:6789/0 conn(0x7f912c142690 legacy=0x7f912c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:39.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.074+0000 7f912a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:03:39.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.074+0000 7f912a7fc640 1 monclient: found mon.noname-b 2024-03-29T11:03:39.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.074+0000 7f912a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:39.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.074+0000 7f912a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:03:39.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.074+0000 7f912a7fc640 1 -- 172.21.15.71:0/4294566903 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f912c151940 con 0x7f912c0b17d0 2024-03-29T11:03:39.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.074+0000 7f912a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:03:39.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.074+0000 7f912a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:03:39.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.074+0000 7f912a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:39.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.074+0000 7f912a7fc640 1 -- 172.21.15.71:0/4294566903 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f91240019f0 con 0x7f912c0b17d0 2024-03-29T11:03:39.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.074+0000 7f9131ecf640 5 monclient: authenticate success, global_id 4352 2024-03-29T11:03:39.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.074+0000 7f912a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:39.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.074+0000 7f912a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:03:39.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.074+0000 7f912a7fc640 10 monclient: dump: 2024-03-29T11:03:39.059 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:39.059 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:39.059 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:39.059 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:39.059 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:39.059 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:39.059 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:39.060 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:39.060 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:39.060 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:39.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.074+0000 7f9131ecf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:39.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.074+0000 7f912a7fc640 1 -- 172.21.15.71:0/4294566903 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9124001b50 con 0x7f912c0b17d0 2024-03-29T11:03:39.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.074+0000 7f912a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:39.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.074+0000 7f912a7fc640 1 -- 172.21.15.71:0/4294566903 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9124001ee0 con 0x7f912c0b17d0 2024-03-29T11:03:39.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.074+0000 7f912a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:39.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.074+0000 7f912a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:39.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.074+0000 7f912a7fc640 10 monclient: dump: 2024-03-29T11:03:39.060 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:39.060 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:39.060 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:39.060 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:39.060 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:39.061 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:39.061 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:39.061 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:39.061 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:39.061 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:39.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.074+0000 7f9131ecf640 10 monclient: get_monmap_and_config success 2024-03-29T11:03:39.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.074+0000 7f9131ecf640 4 set_mon_vals no callback set 2024-03-29T11:03:39.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.075+0000 7f9131ecf640 10 monclient: shutdown 2024-03-29T11:03:39.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.075+0000 7f9131ecf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:39.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.075+0000 7f9131ecf640 1 -- 172.21.15.71:0/4294566903 >> v1:172.21.15.71:6790/0 conn(0x7f912c0b17d0 legacy=0x7f912c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:39.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.076+0000 7f9131ecf640 1 -- 172.21.15.71:0/4294566903 shutdown_connections 2024-03-29T11:03:39.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.076+0000 7f9131ecf640 1 -- 172.21.15.71:0/4294566903 >> 172.21.15.71:0/4294566903 conn(0x7f912c066a00 msgr2=0x7f912c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:39.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.076+0000 7f9131ecf640 1 -- 172.21.15.71:0/4294566903 shutdown_connections 2024-03-29T11:03:39.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.076+0000 7f9131ecf640 1 -- 172.21.15.71:0/4294566903 wait complete. 2024-03-29T11:03:39.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.077+0000 7f9131ecf640 10 monclient: build_initial_monmap 2024-03-29T11:03:39.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.077+0000 7f9131ecf640 10 monclient: monmap: 2024-03-29T11:03:39.073 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:03:39.073 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:03:39.073 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:03:39.073 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:03:39.073 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:39.073 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:39.073 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:03:39.073 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:03:39.073 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:03:39.073 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:39.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.078+0000 7f9131ecf640 1 librados: starting msgr at 2024-03-29T11:03:39.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.078+0000 7f9131ecf640 1 librados: starting objecter 2024-03-29T11:03:39.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.078+0000 7f9131ecf640 1 Processor -- start 2024-03-29T11:03:39.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.079+0000 7f9131ecf640 1 -- start start 2024-03-29T11:03:39.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.079+0000 7f9131ecf640 1 librados: setting wanted keys 2024-03-29T11:03:39.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.079+0000 7f9131ecf640 1 librados: calling monclient init 2024-03-29T11:03:39.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.079+0000 7f9131ecf640 10 monclient: init 2024-03-29T11:03:39.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.079+0000 7f9131ecf640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:39.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.079+0000 7f9131ecf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:03:39.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.079+0000 7f9131ecf640 10 monclient(hunting): picked mon.noname-c con 0x7f912c0b17d0 addr v1:172.21.15.71:6790/0 2024-03-29T11:03:39.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.083+0000 7f9131ecf640 10 monclient(hunting): picked mon.noname-b con 0x7f912c1422b0 addr v1:172.21.15.137:6789/0 2024-03-29T11:03:39.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.085+0000 7f912b7fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f912c0b17d0 0x7f912c07f110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:54687/0 (socket says 172.21.15.71:54687) 2024-03-29T11:03:39.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.086+0000 7f912affd640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f912c1422b0 0x7f912c0b0980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:52251/0 (socket says 172.21.15.71:52251) 2024-03-29T11:03:39.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.086+0000 7f912bfff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f912c142690 0x7f912c087490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:35111/0 (socket says 172.21.15.71:35111) 2024-03-29T11:03:39.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.086+0000 7f9131ecf640 10 monclient(hunting): picked mon.noname-a con 0x7f912c142690 addr v1:172.21.15.71:6789/0 2024-03-29T11:03:39.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.086+0000 7f9131ecf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f912c0b7f80 con 0x7f912c142690 2024-03-29T11:03:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.086+0000 7f9131ecf640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f912c132bf0 con 0x7f912c1422b0 2024-03-29T11:03:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.086+0000 7f9131ecf640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f912c132240 con 0x7f912c0b17d0 2024-03-29T11:03:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.086+0000 7f9131ecf640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.086+0000 7f9131ecf640 10 monclient(hunting): authenticate will time out at 975.345760s 2024-03-29T11:03:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.086+0000 7f912b7fe640 1 -- 172.21.15.71:0/3021551841 learned_addr learned my addr 172.21.15.71:0/3021551841 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:03:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 client.?.objecter ms_handle_connect 0x7f912c1422b0 2024-03-29T11:03:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 client.?.objecter ms_handle_connect 0x7f912c142690 2024-03-29T11:03:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 client.?.objecter ms_handle_connect 0x7f912c0b17d0 2024-03-29T11:03:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 1 -- 172.21.15.71:0/3021551841 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 387995925 0 0) 0x7f912c132bf0 con 0x7f912c1422b0 2024-03-29T11:03:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 monclient(hunting): _negotiate 2024-03-29T11:03:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:39.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:03:39.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 1 -- 172.21.15.71:0/3021551841 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9118004eb0 con 0x7f912c1422b0 2024-03-29T11:03:39.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 1 -- 172.21.15.71:0/3021551841 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2515338352 0 0) 0x7f912c0b7f80 con 0x7f912c142690 2024-03-29T11:03:39.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 monclient(hunting): _negotiate 2024-03-29T11:03:39.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:39.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:39.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 monclient(hunting): my global_id is 4356 2024-03-29T11:03:39.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 1 -- 172.21.15.71:0/3021551841 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f912c132bf0 con 0x7f912c142690 2024-03-29T11:03:39.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 1 -- 172.21.15.71:0/3021551841 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153778201 0 0) 0x7f912c132240 con 0x7f912c0b17d0 2024-03-29T11:03:39.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 monclient(hunting): _negotiate 2024-03-29T11:03:39.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:39.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:39.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:03:39.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 1 -- 172.21.15.71:0/3021551841 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f912c0b7f80 con 0x7f912c0b17d0 2024-03-29T11:03:39.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 1 -- 172.21.15.71:0/3021551841 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1751307538 0 0) 0x7f9118004eb0 con 0x7f912c1422b0 2024-03-29T11:03:39.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 1 -- 172.21.15.71:0/3021551841 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f912c132240 con 0x7f912c1422b0 2024-03-29T11:03:39.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 1 -- 172.21.15.71:0/3021551841 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f91200018f0 con 0x7f912c1422b0 2024-03-29T11:03:39.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:39.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:03:39.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 monclient(hunting): dump: 2024-03-29T11:03:39.077 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:39.077 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:39.077 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:39.077 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:39.077 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:39.077 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:39.077 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:39.078 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:39.078 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:39.078 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:39.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:03:39.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 1 -- 172.21.15.71:0/3021551841 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3330163923 0 0) 0x7f912c132bf0 con 0x7f912c142690 2024-03-29T11:03:39.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 1 -- 172.21.15.71:0/3021551841 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9118004eb0 con 0x7f912c142690 2024-03-29T11:03:39.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 1 -- 172.21.15.71:0/3021551841 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9124001cc0 con 0x7f912c142690 2024-03-29T11:03:39.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:39.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:03:39.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 monclient(hunting): dump: 2024-03-29T11:03:39.078 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:39.078 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:39.079 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:39.079 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:39.079 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:39.079 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:39.079 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:39.079 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:39.079 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:39.079 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:39.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 1 -- 172.21.15.71:0/3021551841 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 135403188 0 0) 0x7f912c0b7f80 con 0x7f912c0b17d0 2024-03-29T11:03:39.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 1 -- 172.21.15.71:0/3021551841 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f912c132bf0 con 0x7f912c0b17d0 2024-03-29T11:03:39.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 1 -- 172.21.15.71:0/3021551841 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f911c001890 con 0x7f912c0b17d0 2024-03-29T11:03:39.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:39.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:39.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 10 monclient(hunting): dump: 2024-03-29T11:03:39.080 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:39.080 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:39.080 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:39.080 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:39.080 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:39.080 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:39.080 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:39.080 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:39.080 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:39.080 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:39.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.087+0000 7f9113fff640 1 -- 172.21.15.71:0/3021551841 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 419951609 0 0) 0x7f912c132240 con 0x7f912c1422b0 2024-03-29T11:03:39.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.088+0000 7f9113fff640 1 -- 172.21.15.71:0/3021551841 >> v1:172.21.15.71:6790/0 conn(0x7f912c0b17d0 legacy=0x7f912c07f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:39.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.088+0000 7f9113fff640 1 -- 172.21.15.71:0/3021551841 >> v1:172.21.15.71:6789/0 conn(0x7f912c142690 legacy=0x7f912c087490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:39.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.088+0000 7f9113fff640 10 monclient: _finish_hunting 0 2024-03-29T11:03:39.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.088+0000 7f9113fff640 1 monclient: found mon.b 2024-03-29T11:03:39.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.088+0000 7f9113fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:39.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.088+0000 7f9113fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:03:39.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.088+0000 7f9113fff640 1 -- 172.21.15.71:0/3021551841 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f912c1323d0 con 0x7f912c1422b0 2024-03-29T11:03:39.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.088+0000 7f9113fff640 10 monclient: _finish_auth 0 2024-03-29T11:03:39.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.088+0000 7f9113fff640 10 monclient: _check_auth_tickets 2024-03-29T11:03:39.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.088+0000 7f9113fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:39.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.088+0000 7f9131ecf640 5 monclient: authenticate success, global_id 4312 2024-03-29T11:03:39.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.088+0000 7f9131ecf640 10 monclient: _renew_subs 2024-03-29T11:03:39.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.088+0000 7f9131ecf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:03:39.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.088+0000 7f9131ecf640 1 -- 172.21.15.71:0/3021551841 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f912c07f820 con 0x7f912c1422b0 2024-03-29T11:03:39.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.088+0000 7f9131ecf640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:03:39.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.088+0000 7f9131ecf640 10 monclient: _renew_subs 2024-03-29T11:03:39.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.088+0000 7f9131ecf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:03:39.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.088+0000 7f9131ecf640 1 -- 172.21.15.71:0/3021551841 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f912c0b85f0 con 0x7f912c1422b0 2024-03-29T11:03:39.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.088+0000 7f9131ecf640 1 librados: init done 2024-03-29T11:03:39.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.088+0000 7f9113fff640 1 -- 172.21.15.71:0/3021551841 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9120001a50 con 0x7f912c1422b0 2024-03-29T11:03:39.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.088+0000 7f9113fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:39.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.088+0000 7f9113fff640 1 -- 172.21.15.71:0/3021551841 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f91200037b0 con 0x7f912c1422b0 2024-03-29T11:03:39.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.088+0000 7f9113fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:39.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.088+0000 7f9129ffb640 4 set_mon_vals no callback set 2024-03-29T11:03:39.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.088+0000 7f9113fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:03:39.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.088+0000 7f9113fff640 10 monclient: dump: 2024-03-29T11:03:39.085 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:39.085 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:39.085 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:39.085 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:39.085 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:39.085 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:39.085 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:39.086 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:39.086 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:39.086 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:39.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.089+0000 7f9131ecf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:39.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.089+0000 7f9131ecf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:39.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.089+0000 7f9131ecf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:03:39.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.089+0000 7f9131ecf640 1 -- 172.21.15.71:0/3021551841 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f910c005080 con 0x7f912c1422b0 2024-03-29T11:03:39.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.095+0000 7f9113fff640 1 -- 172.21.15.71:0/3021551841 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f9120003910 con 0x7f912c1422b0 2024-03-29T11:03:39.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.096+0000 7f9113fff640 1 -- 172.21.15.71:0/3021551841 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f91200848e0 con 0x7f912c1422b0 2024-03-29T11:03:39.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.096+0000 7f9113fff640 10 client.4312.objecter ms_dispatch 0x7f912c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:03:39.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.096+0000 7f9113fff640 3 client.4312.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:03:39.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.096+0000 7f9113fff640 3 client.4312.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:03:39.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.096+0000 7f9113fff640 20 client.4312.objecter dump_active .. 0 homeless 2024-03-29T11:03:39.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.096+0000 7f912b7fe640 10 monclient: get_auth_request con 0x7f911806e170 auth_method 0 2024-03-29T11:03:39.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.097+0000 7f9113fff640 10 client.4312.objecter ms_handle_connect 0x7f911806e170 2024-03-29T11:03:39.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.101+0000 7f9113fff640 1 -- 172.21.15.71:0/3021551841 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f912004fff0 con 0x7f912c1422b0 2024-03-29T11:03:39.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.101+0000 7f9113fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:39.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.101+0000 7f9113fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:03:39.400 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.420+0000 7f9131ecf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:39.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.420+0000 7f9131ecf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:39.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.420+0000 7f9131ecf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:03:39.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.420+0000 7f9131ecf640 1 -- 172.21.15.71:0/3021551841 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f910c005330 con 0x7f912c1422b0 2024-03-29T11:03:39.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.423+0000 7f9113fff640 1 -- 172.21.15.71:0/3021551841 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f91200520f0 con 0x7f912c1422b0 2024-03-29T11:03:39.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.423+0000 7f9113fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:39.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.423+0000 7f9113fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:03:39.404 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:03:39.404 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:03:39.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.427+0000 7f9131ecf640 10 librados: watch_flush enter 2024-03-29T11:03:39.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.431+0000 7f9131ecf640 10 librados: watch_flush exit 2024-03-29T11:03:39.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.431+0000 7f9131ecf640 20 client.4312.objecter shutdown clearing up homeless session... 2024-03-29T11:03:39.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.431+0000 7f9131ecf640 10 client.4312.objecter successfully canceled tick 2024-03-29T11:03:39.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.431+0000 7f9131ecf640 1 -- 172.21.15.71:0/3021551841 >> v1:172.21.15.137:6816/973332665 conn(0x7f911806e170 legacy=0x7f9118070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:39.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.431+0000 7f9131ecf640 10 monclient: shutdown 2024-03-29T11:03:39.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.431+0000 7f9131ecf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:39.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.431+0000 7f9131ecf640 1 -- 172.21.15.71:0/3021551841 >> v1:172.21.15.137:6789/0 conn(0x7f912c1422b0 legacy=0x7f912c0b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:39.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.432+0000 7f9131ecf640 1 -- 172.21.15.71:0/3021551841 shutdown_connections 2024-03-29T11:03:39.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.432+0000 7f9131ecf640 1 -- 172.21.15.71:0/3021551841 >> 172.21.15.71:0/3021551841 conn(0x7f912c066a00 msgr2=0x7f912c13e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:39.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.433+0000 7f9131ecf640 1 -- 172.21.15.71:0/3021551841 shutdown_connections 2024-03-29T11:03:39.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.433+0000 7f9131ecf640 1 -- 172.21.15.71:0/3021551841 wait complete. 2024-03-29T11:03:39.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:39.435+0000 7f9131ecf640 1 librados: shutdown 2024-03-29T11:03:39.432 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:03:46.238 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:03:46.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.405+0000 7fae9a8ae640 10 monclient: get_monmap_and_config 2024-03-29T11:03:46.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.405+0000 7fae9a8ae640 10 monclient: build_initial_monmap 2024-03-29T11:03:46.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.405+0000 7fae9a8ae640 10 monclient: monmap: 2024-03-29T11:03:46.386 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:03:46.386 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:46.386 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:03:46.407479+0000 2024-03-29T11:03:46.386 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:03:46.407479+0000 2024-03-29T11:03:46.386 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:46.386 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:46.386 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:03:46.387 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:03:46.387 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:03:46.387 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:46.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.407+0000 7fae9a8ae640 1 Processor -- start 2024-03-29T11:03:46.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.407+0000 7fae9a8ae640 1 -- start start 2024-03-29T11:03:46.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.407+0000 7fae9a8ae640 10 monclient: init 2024-03-29T11:03:46.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.408+0000 7fae9a8ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:46.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.408+0000 7fae9a8ae640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:03:46.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.408+0000 7fae9a8ae640 10 monclient(hunting): picked mon.noname-c con 0x7fae8c038c60 addr v1:172.21.15.137:6789/0 2024-03-29T11:03:46.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.408+0000 7fae9a8ae640 10 monclient(hunting): picked mon.noname-a con 0x7fae8c0324b0 addr v1:172.21.15.71:6789/0 2024-03-29T11:03:46.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.411+0000 7fae9a8ae640 10 monclient(hunting): picked mon.noname-b con 0x7fae8c0d11e0 addr v1:172.21.15.71:6790/0 2024-03-29T11:03:46.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.411+0000 7fae998ac640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fae8c038c60 0x7fae8c0c4ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:45457/0 (socket says 172.21.15.71:45457) 2024-03-29T11:03:46.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.411+0000 7fae998ac640 1 -- 172.21.15.71:0/1832227155 learned_addr learned my addr 172.21.15.71:0/1832227155 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:03:46.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.411+0000 7fae9a8ae640 1 -- 172.21.15.71:0/1832227155 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c0caf80 con 0x7fae8c0324b0 2024-03-29T11:03:46.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.412+0000 7fae9a8ae640 1 -- 172.21.15.71:0/1832227155 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c039850 con 0x7fae8c038c60 2024-03-29T11:03:46.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.412+0000 7fae9a8ae640 1 -- 172.21.15.71:0/1832227155 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c0c78d0 con 0x7fae8c0d11e0 2024-03-29T11:03:46.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.412+0000 7fae9a8ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:46.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.412+0000 7fae9a8ae640 10 monclient(hunting): authenticate will time out at 982.671426s 2024-03-29T11:03:46.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.414+0000 7fae988aa640 1 -- 172.21.15.71:0/1832227155 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113107867 0 0) 0x7fae8c0c78d0 con 0x7fae8c0d11e0 2024-03-29T11:03:46.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.414+0000 7fae988aa640 10 monclient(hunting): _negotiate 2024-03-29T11:03:46.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.414+0000 7fae988aa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:46.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.414+0000 7fae988aa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:46.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.414+0000 7fae988aa640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:03:46.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.414+0000 7fae988aa640 1 -- 172.21.15.71:0/1832227155 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae80003520 con 0x7fae8c0d11e0 2024-03-29T11:03:46.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.415+0000 7fae988aa640 1 -- 172.21.15.71:0/1832227155 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565702978 0 0) 0x7fae8c039850 con 0x7fae8c038c60 2024-03-29T11:03:46.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.415+0000 7fae988aa640 10 monclient(hunting): _negotiate 2024-03-29T11:03:46.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.415+0000 7fae988aa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:46.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.415+0000 7fae988aa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:46.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.415+0000 7fae988aa640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:03:46.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.415+0000 7fae988aa640 1 -- 172.21.15.71:0/1832227155 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae8c0c78d0 con 0x7fae8c038c60 2024-03-29T11:03:46.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.415+0000 7fae988aa640 1 -- 172.21.15.71:0/1832227155 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1466920384 0 0) 0x7fae80003520 con 0x7fae8c0d11e0 2024-03-29T11:03:46.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.415+0000 7fae988aa640 1 -- 172.21.15.71:0/1832227155 >> v1:172.21.15.137:6789/0 conn(0x7fae8c038c60 legacy=0x7fae8c0c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:46.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.415+0000 7fae988aa640 1 -- 172.21.15.71:0/1832227155 >> v1:172.21.15.71:6789/0 conn(0x7fae8c0324b0 legacy=0x7fae8c0cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:46.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.415+0000 7fae988aa640 10 monclient: _finish_hunting 0 2024-03-29T11:03:46.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.416+0000 7fae988aa640 1 monclient: found mon.noname-b 2024-03-29T11:03:46.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.416+0000 7fae988aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:46.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.416+0000 7fae988aa640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:03:46.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.416+0000 7fae988aa640 1 -- 172.21.15.71:0/1832227155 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c0c7a60 con 0x7fae8c0d11e0 2024-03-29T11:03:46.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.416+0000 7fae988aa640 10 monclient: _finish_auth 0 2024-03-29T11:03:46.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.416+0000 7fae988aa640 10 monclient: _check_auth_tickets 2024-03-29T11:03:46.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.416+0000 7fae988aa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:46.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.416+0000 7fae988aa640 1 -- 172.21.15.71:0/1832227155 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fae90001690 con 0x7fae8c0d11e0 2024-03-29T11:03:46.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.416+0000 7fae9a8ae640 5 monclient: authenticate success, global_id 4358 2024-03-29T11:03:46.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.416+0000 7fae9a8ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:46.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.416+0000 7fae988aa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:46.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.416+0000 7fae988aa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:03:46.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.416+0000 7fae988aa640 10 monclient: dump: 2024-03-29T11:03:46.399 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:46.399 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:46.399 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:46.399 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:46.399 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:46.399 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:46.399 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:46.399 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:46.399 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:46.399 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:46.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.416+0000 7fae9a8ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:46.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.419+0000 7fae988aa640 1 -- 172.21.15.71:0/1832227155 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae90001860 con 0x7fae8c0d11e0 2024-03-29T11:03:46.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.419+0000 7fae988aa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:46.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.419+0000 7fae988aa640 1 -- 172.21.15.71:0/1832227155 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fae90001cb0 con 0x7fae8c0d11e0 2024-03-29T11:03:46.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.419+0000 7fae988aa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:46.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.419+0000 7fae988aa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:46.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.419+0000 7fae988aa640 10 monclient: dump: 2024-03-29T11:03:46.401 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:46.401 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:46.401 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:46.402 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:46.402 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:46.402 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:46.402 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:46.402 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:46.402 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:46.402 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:46.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.419+0000 7fae9a8ae640 10 monclient: get_monmap_and_config success 2024-03-29T11:03:46.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.419+0000 7fae9a8ae640 4 set_mon_vals no callback set 2024-03-29T11:03:46.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.420+0000 7fae9a8ae640 10 monclient: shutdown 2024-03-29T11:03:46.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.420+0000 7fae9a8ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:46.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.420+0000 7fae9a8ae640 1 -- 172.21.15.71:0/1832227155 >> v1:172.21.15.71:6790/0 conn(0x7fae8c0d11e0 legacy=0x7fae8c0d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:46.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.420+0000 7fae9a8ae640 1 -- 172.21.15.71:0/1832227155 shutdown_connections 2024-03-29T11:03:46.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.420+0000 7fae9a8ae640 1 -- 172.21.15.71:0/1832227155 >> 172.21.15.71:0/1832227155 conn(0x7fae8c005c20 msgr2=0x7fae8c006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:46.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.421+0000 7fae9a8ae640 1 -- 172.21.15.71:0/1832227155 shutdown_connections 2024-03-29T11:03:46.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.421+0000 7fae9a8ae640 1 -- 172.21.15.71:0/1832227155 wait complete. 2024-03-29T11:03:46.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.423+0000 7fae9a8ae640 10 monclient: build_initial_monmap 2024-03-29T11:03:46.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.423+0000 7fae9a8ae640 10 monclient: monmap: 2024-03-29T11:03:46.404 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:03:46.404 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:03:46.404 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:03:46.404 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:03:46.404 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:46.404 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:46.404 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:03:46.405 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:03:46.405 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:03:46.405 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:46.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.423+0000 7fae9a8ae640 1 librados: starting msgr at 2024-03-29T11:03:46.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.424+0000 7fae9a8ae640 1 librados: starting objecter 2024-03-29T11:03:46.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.424+0000 7fae9a8ae640 1 Processor -- start 2024-03-29T11:03:46.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.424+0000 7fae9a8ae640 1 -- start start 2024-03-29T11:03:46.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.425+0000 7fae9a8ae640 1 librados: setting wanted keys 2024-03-29T11:03:46.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.425+0000 7fae9a8ae640 1 librados: calling monclient init 2024-03-29T11:03:46.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.425+0000 7fae9a8ae640 10 monclient: init 2024-03-29T11:03:46.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.427+0000 7fae9a8ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:46.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.427+0000 7fae9a8ae640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:03:46.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.427+0000 7fae9a8ae640 10 monclient(hunting): picked mon.noname-c con 0x7fae8c0324b0 addr v1:172.21.15.71:6790/0 2024-03-29T11:03:46.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.427+0000 7fae9a8ae640 10 monclient(hunting): picked mon.noname-a con 0x7fae8c038c60 addr v1:172.21.15.71:6789/0 2024-03-29T11:03:46.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.429+0000 7fae9a8ae640 10 monclient(hunting): picked mon.noname-b con 0x7fae8c0d11e0 addr v1:172.21.15.137:6789/0 2024-03-29T11:03:46.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.429+0000 7fae9a8ae640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c035ab0 con 0x7fae8c038c60 2024-03-29T11:03:46.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.429+0000 7fae9a8ae640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c0369c0 con 0x7fae8c0d11e0 2024-03-29T11:03:46.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.429+0000 7fae9a8ae640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c036c30 con 0x7fae8c0324b0 2024-03-29T11:03:46.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.429+0000 7fae9a8ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:46.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.429+0000 7fae9a8ae640 10 monclient(hunting): authenticate will time out at 982.689049s 2024-03-29T11:03:46.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.431+0000 7fae990ab640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fae8c038c60 0x7fae8c0be210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:39665/0 (socket says 172.21.15.71:39665) 2024-03-29T11:03:46.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.431+0000 7fae990ab640 1 -- 172.21.15.71:0/1071651958 learned_addr learned my addr 172.21.15.71:0/1071651958 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:03:46.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.432+0000 7fae8a7fc640 10 client.?.objecter ms_handle_connect 0x7fae8c038c60 2024-03-29T11:03:46.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.432+0000 7fae8a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:46.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.432+0000 7fae8a7fc640 10 client.?.objecter ms_handle_connect 0x7fae8c0324b0 2024-03-29T11:03:46.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.432+0000 7fae8a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:46.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.432+0000 7fae8a7fc640 10 client.?.objecter ms_handle_connect 0x7fae8c0d11e0 2024-03-29T11:03:46.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.432+0000 7fae8a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:46.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.432+0000 7fae8a7fc640 1 -- 172.21.15.71:0/1071651958 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1369311505 0 0) 0x7fae8c035ab0 con 0x7fae8c038c60 2024-03-29T11:03:46.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.432+0000 7fae8a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:03:46.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.432+0000 7fae8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:46.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.432+0000 7fae8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:46.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.432+0000 7fae8a7fc640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:03:46.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.432+0000 7fae8a7fc640 1 -- 172.21.15.71:0/1071651958 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae78004eb0 con 0x7fae8c038c60 2024-03-29T11:03:46.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.432+0000 7fae8a7fc640 1 -- 172.21.15.71:0/1071651958 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1311170732 0 0) 0x7fae78004eb0 con 0x7fae8c038c60 2024-03-29T11:03:46.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.432+0000 7fae8a7fc640 1 -- 172.21.15.71:0/1071651958 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae8c035ab0 con 0x7fae8c038c60 2024-03-29T11:03:46.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.432+0000 7fae8a7fc640 1 -- 172.21.15.71:0/1071651958 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fae84001a10 con 0x7fae8c038c60 2024-03-29T11:03:46.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.432+0000 7fae8a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:46.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.432+0000 7fae8a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:03:46.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.432+0000 7fae8a7fc640 10 monclient(hunting): dump: 2024-03-29T11:03:46.418 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:46.418 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:46.418 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:46.418 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:46.418 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:46.418 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:46.418 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:46.418 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:46.418 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:46.418 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:46.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.432+0000 7fae8a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:03:46.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.433+0000 7fae8a7fc640 1 -- 172.21.15.71:0/1071651958 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1383429075 0 0) 0x7fae8c035ab0 con 0x7fae8c038c60 2024-03-29T11:03:46.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.433+0000 7fae8a7fc640 1 -- 172.21.15.71:0/1071651958 >> v1:172.21.15.71:6790/0 conn(0x7fae8c0324b0 legacy=0x7fae8c0353a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:46.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.433+0000 7fae8a7fc640 1 -- 172.21.15.71:0/1071651958 >> v1:172.21.15.137:6789/0 conn(0x7fae8c0d11e0 legacy=0x7fae8c15cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:46.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.433+0000 7fae8a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:03:46.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.433+0000 7fae8a7fc640 1 monclient: found mon.a 2024-03-29T11:03:46.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.433+0000 7fae8a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:46.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.433+0000 7fae8a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:03:46.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.433+0000 7fae8a7fc640 1 -- 172.21.15.71:0/1071651958 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c034700 con 0x7fae8c038c60 2024-03-29T11:03:46.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.433+0000 7fae8a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:03:46.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.433+0000 7fae8a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:03:46.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.433+0000 7fae8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:46.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.433+0000 7fae9a8ae640 5 monclient: authenticate success, global_id 4362 2024-03-29T11:03:46.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.433+0000 7fae9a8ae640 10 monclient: _renew_subs 2024-03-29T11:03:46.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.433+0000 7fae9a8ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:03:46.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.433+0000 7fae9a8ae640 1 -- 172.21.15.71:0/1071651958 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae8c037050 con 0x7fae8c038c60 2024-03-29T11:03:46.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.433+0000 7fae9a8ae640 10 client.4362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:03:46.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.433+0000 7fae9a8ae640 10 monclient: _renew_subs 2024-03-29T11:03:46.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.433+0000 7fae9a8ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:03:46.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.433+0000 7fae9a8ae640 1 -- 172.21.15.71:0/1071651958 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae8c0beb00 con 0x7fae8c038c60 2024-03-29T11:03:46.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.433+0000 7fae9a8ae640 1 librados: init done 2024-03-29T11:03:46.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.435+0000 7fae9a8ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:46.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.435+0000 7fae9a8ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:46.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.435+0000 7fae9a8ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:03:46.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.435+0000 7fae9a8ae640 1 -- 172.21.15.71:0/1071651958 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae8c007b90 con 0x7fae8c038c60 2024-03-29T11:03:46.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.435+0000 7fae8a7fc640 1 -- 172.21.15.71:0/1071651958 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae84001cd0 con 0x7fae8c038c60 2024-03-29T11:03:46.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.435+0000 7fae8a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:46.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.435+0000 7fae8bfff640 4 set_mon_vals no callback set 2024-03-29T11:03:46.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.435+0000 7fae8a7fc640 1 -- 172.21.15.71:0/1071651958 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fae84003a60 con 0x7fae8c038c60 2024-03-29T11:03:46.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.435+0000 7fae8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:46.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.435+0000 7fae8a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:03:46.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.435+0000 7fae8a7fc640 10 monclient: dump: 2024-03-29T11:03:46.421 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:46.421 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:46.421 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:46.421 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:46.421 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:46.421 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:46.421 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:46.421 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:46.421 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:46.421 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:46.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.437+0000 7fae8a7fc640 1 -- 172.21.15.71:0/1071651958 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7fae84003bc0 con 0x7fae8c038c60 2024-03-29T11:03:46.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.438+0000 7fae998ac640 10 monclient: get_auth_request con 0x7fae7806dd80 auth_method 0 2024-03-29T11:03:46.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.439+0000 7fae8a7fc640 1 -- 172.21.15.71:0/1071651958 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7fae84084ae0 con 0x7fae8c038c60 2024-03-29T11:03:46.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.439+0000 7fae8a7fc640 10 client.4362.objecter ms_dispatch 0x7fae94000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:03:46.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.439+0000 7fae8a7fc640 3 client.4362.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:03:46.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.439+0000 7fae8a7fc640 3 client.4362.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:03:46.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.439+0000 7fae8a7fc640 20 client.4362.objecter dump_active .. 0 homeless 2024-03-29T11:03:46.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.439+0000 7fae8a7fc640 10 client.4362.objecter ms_handle_connect 0x7fae7806dd80 2024-03-29T11:03:46.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.443+0000 7fae8a7fc640 1 -- 172.21.15.71:0/1071651958 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fae840503c0 con 0x7fae8c038c60 2024-03-29T11:03:46.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.443+0000 7fae8a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:46.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.443+0000 7fae8a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:03:46.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.768+0000 7fae9a8ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:46.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.768+0000 7fae9a8ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:46.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.768+0000 7fae9a8ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:03:46.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.768+0000 7fae9a8ae640 1 -- 172.21.15.71:0/1071651958 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae8c004cd0 con 0x7fae8c038c60 2024-03-29T11:03:46.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.769+0000 7fae8a7fc640 1 -- 172.21.15.71:0/1071651958 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7fae840522f0 con 0x7fae8c038c60 2024-03-29T11:03:46.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.769+0000 7fae8a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:46.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.769+0000 7fae8a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:03:46.752 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:03:46.752 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:03:46.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.774+0000 7fae9a8ae640 10 librados: watch_flush enter 2024-03-29T11:03:46.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.775+0000 7fae9a8ae640 10 librados: watch_flush exit 2024-03-29T11:03:46.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.775+0000 7fae9a8ae640 20 client.4362.objecter shutdown clearing up homeless session... 2024-03-29T11:03:46.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.775+0000 7fae9a8ae640 10 client.4362.objecter successfully canceled tick 2024-03-29T11:03:46.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.776+0000 7fae9a8ae640 1 -- 172.21.15.71:0/1071651958 >> v1:172.21.15.137:6816/973332665 conn(0x7fae7806dd80 legacy=0x7fae78070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:46.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.776+0000 7fae9a8ae640 10 monclient: shutdown 2024-03-29T11:03:46.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.776+0000 7fae9a8ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:46.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.776+0000 7fae9a8ae640 1 -- 172.21.15.71:0/1071651958 >> v1:172.21.15.71:6789/0 conn(0x7fae8c038c60 legacy=0x7fae8c0be210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:46.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.777+0000 7fae9a8ae640 1 -- 172.21.15.71:0/1071651958 shutdown_connections 2024-03-29T11:03:46.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.777+0000 7fae9a8ae640 1 -- 172.21.15.71:0/1071651958 >> 172.21.15.71:0/1071651958 conn(0x7fae8c005c20 msgr2=0x7fae8c0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:46.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.779+0000 7fae9a8ae640 1 -- 172.21.15.71:0/1071651958 shutdown_connections 2024-03-29T11:03:46.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.779+0000 7fae9a8ae640 1 -- 172.21.15.71:0/1071651958 wait complete. 2024-03-29T11:03:46.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:46.779+0000 7fae9a8ae640 1 librados: shutdown 2024-03-29T11:03:46.774 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:03:53.581 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:03:53.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.771+0000 7fb1af59e640 10 monclient: get_monmap_and_config 2024-03-29T11:03:53.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.771+0000 7fb1af59e640 10 monclient: build_initial_monmap 2024-03-29T11:03:53.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.771+0000 7fb1af59e640 10 monclient: monmap: 2024-03-29T11:03:53.751 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:03:53.751 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:53.751 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:03:53.772005+0000 2024-03-29T11:03:53.751 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:03:53.772005+0000 2024-03-29T11:03:53.751 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:53.751 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:53.751 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:03:53.751 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:03:53.751 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:03:53.751 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:53.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.772+0000 7fb1af59e640 1 Processor -- start 2024-03-29T11:03:53.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.772+0000 7fb1af59e640 1 -- start start 2024-03-29T11:03:53.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.772+0000 7fb1af59e640 10 monclient: init 2024-03-29T11:03:53.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.773+0000 7fb1af59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:53.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.773+0000 7fb1af59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:03:53.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.773+0000 7fb1af59e640 10 monclient(hunting): picked mon.noname-b con 0x7fb1b00b4820 addr v1:172.21.15.71:6790/0 2024-03-29T11:03:53.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.773+0000 7fb1af59e640 10 monclient(hunting): picked mon.noname-c con 0x7fb1b00a9ce0 addr v1:172.21.15.137:6789/0 2024-03-29T11:03:53.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.773+0000 7fb1af59e640 10 monclient(hunting): picked mon.noname-a con 0x7fb1b00b3cd0 addr v1:172.21.15.71:6789/0 2024-03-29T11:03:53.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.773+0000 7fb1af59e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1b00b4180 con 0x7fb1b00b3cd0 2024-03-29T11:03:53.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.773+0000 7fb1af59e640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1b00aaf60 con 0x7fb1b00a9ce0 2024-03-29T11:03:53.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.773+0000 7fb1af59e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1b013c200 con 0x7fb1b00b4820 2024-03-29T11:03:53.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.773+0000 7fb1af59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:53.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.773+0000 7fb1af59e640 10 monclient(hunting): authenticate will time out at 990.031494s 2024-03-29T11:03:53.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.773+0000 7fb1add9b640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fb1b00a9ce0 0x7fb1b0145770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:49320/0 (socket says 172.21.15.71:49320) 2024-03-29T11:03:53.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.773+0000 7fb1add9b640 1 -- 172.21.15.71:0/2832211142 learned_addr learned my addr 172.21.15.71:0/2832211142 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:03:53.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 1 -- 172.21.15.71:0/2832211142 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4036466195 0 0) 0x7fb1b00aaf60 con 0x7fb1b00a9ce0 2024-03-29T11:03:53.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 10 monclient(hunting): _negotiate 2024-03-29T11:03:53.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:53.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:53.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 10 monclient(hunting): my global_id is 4321 2024-03-29T11:03:53.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 1 -- 172.21.15.71:0/2832211142 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb198003520 con 0x7fb1b00a9ce0 2024-03-29T11:03:53.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 1 -- 172.21.15.71:0/2832211142 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253518011 0 0) 0x7fb1a4001110 con 0x7fb1b00b3cd0 2024-03-29T11:03:53.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 10 monclient(hunting): _negotiate 2024-03-29T11:03:53.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:53.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:53.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 10 monclient(hunting): my global_id is 4365 2024-03-29T11:03:53.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 1 -- 172.21.15.71:0/2832211142 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1b00aaf60 con 0x7fb1b00b3cd0 2024-03-29T11:03:53.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 1 -- 172.21.15.71:0/2832211142 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2258923712 0 0) 0x7fb1b013c200 con 0x7fb1b00b4820 2024-03-29T11:03:53.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 10 monclient(hunting): _negotiate 2024-03-29T11:03:53.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:53.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:53.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:03:53.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 1 -- 172.21.15.71:0/2832211142 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1a4001110 con 0x7fb1b00b4820 2024-03-29T11:03:53.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 1 -- 172.21.15.71:0/2832211142 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3721026959 0 0) 0x7fb198003520 con 0x7fb1b00a9ce0 2024-03-29T11:03:53.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 1 -- 172.21.15.71:0/2832211142 >> v1:172.21.15.71:6790/0 conn(0x7fb1b00b4820 legacy=0x7fb1b013baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:53.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 1 -- 172.21.15.71:0/2832211142 >> v1:172.21.15.71:6789/0 conn(0x7fb1b00b3cd0 legacy=0x7fb1b014d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:53.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 10 monclient: _finish_hunting 0 2024-03-29T11:03:53.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 1 monclient: found mon.noname-c 2024-03-29T11:03:53.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:53.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.137:6789/0 2024-03-29T11:03:53.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 1 -- 172.21.15.71:0/2832211142 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b0145e80 con 0x7fb1b00a9ce0 2024-03-29T11:03:53.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 10 monclient: _finish_auth 0 2024-03-29T11:03:53.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 10 monclient: _check_auth_tickets 2024-03-29T11:03:53.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.774+0000 7fb1ad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:53.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.775+0000 7fb1ad59a640 1 -- 172.21.15.71:0/2832211142 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb1a80018c0 con 0x7fb1b00a9ce0 2024-03-29T11:03:53.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.775+0000 7fb1ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:53.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.775+0000 7fb1ad59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:03:53.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.775+0000 7fb1ad59a640 10 monclient: dump: 2024-03-29T11:03:53.757 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:53.757 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:53.758 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:53.758 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:53.758 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:53.758 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:53.758 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:53.758 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:53.758 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:53.758 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:53.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.775+0000 7fb1af59e640 5 monclient: authenticate success, global_id 4321 2024-03-29T11:03:53.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.775+0000 7fb1af59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:53.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.775+0000 7fb1ad59a640 1 -- 172.21.15.71:0/2832211142 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1a8001a20 con 0x7fb1b00a9ce0 2024-03-29T11:03:53.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.775+0000 7fb1ad59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:53.761 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.775+0000 7fb1af59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:03:53.761 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.775+0000 7fb1af59e640 4 set_mon_vals no callback set 2024-03-29T11:03:53.761 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.776+0000 7fb1ad59a640 1 -- 172.21.15.71:0/2832211142 <== mon.1 v1:172.21.15.137:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb1a8001e70 con 0x7fb1b00a9ce0 2024-03-29T11:03:53.761 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.776+0000 7fb1ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:53.761 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.776+0000 7fb1ad59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:03:53.761 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.776+0000 7fb1ad59a640 10 monclient: dump: 2024-03-29T11:03:53.761 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:53.762 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:53.762 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:53.762 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:53.762 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:53.762 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:53.762 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:53.762 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:53.762 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:53.762 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:53.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.776+0000 7fb1af59e640 10 monclient: shutdown 2024-03-29T11:03:53.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.776+0000 7fb1af59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:53.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.776+0000 7fb1af59e640 1 -- 172.21.15.71:0/2832211142 >> v1:172.21.15.137:6789/0 conn(0x7fb1b00a9ce0 legacy=0x7fb1b0145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:53.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.776+0000 7fb1af59e640 1 -- 172.21.15.71:0/2832211142 shutdown_connections 2024-03-29T11:03:53.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.776+0000 7fb1af59e640 1 -- 172.21.15.71:0/2832211142 >> 172.21.15.71:0/2832211142 conn(0x7fb1b006d070 msgr2=0x7fb1b0066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:53.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.776+0000 7fb1af59e640 1 -- 172.21.15.71:0/2832211142 shutdown_connections 2024-03-29T11:03:53.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.776+0000 7fb1af59e640 1 -- 172.21.15.71:0/2832211142 wait complete. 2024-03-29T11:03:53.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.778+0000 7fb1af59e640 10 monclient: build_initial_monmap 2024-03-29T11:03:53.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.778+0000 7fb1af59e640 10 monclient: monmap: 2024-03-29T11:03:53.763 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:03:53.763 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:03:53.763 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:03:53.763 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:03:53.763 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:53.763 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:53.763 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:03:53.763 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:03:53.763 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:03:53.763 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:53.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.778+0000 7fb1af59e640 1 librados: starting msgr at 2024-03-29T11:03:53.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.778+0000 7fb1af59e640 1 librados: starting objecter 2024-03-29T11:03:53.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.778+0000 7fb1af59e640 1 Processor -- start 2024-03-29T11:03:53.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.778+0000 7fb1af59e640 1 -- start start 2024-03-29T11:03:53.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.778+0000 7fb1af59e640 1 librados: setting wanted keys 2024-03-29T11:03:53.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.778+0000 7fb1af59e640 1 librados: calling monclient init 2024-03-29T11:03:53.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.778+0000 7fb1af59e640 10 monclient: init 2024-03-29T11:03:53.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.779+0000 7fb1af59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:53.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.779+0000 7fb1af59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:03:53.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.779+0000 7fb1af59e640 10 monclient(hunting): picked mon.noname-b con 0x7fb1b00a9ce0 addr v1:172.21.15.137:6789/0 2024-03-29T11:03:53.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.779+0000 7fb1af59e640 10 monclient(hunting): picked mon.noname-a con 0x7fb1b00b3cd0 addr v1:172.21.15.71:6789/0 2024-03-29T11:03:53.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.779+0000 7fb1af59e640 10 monclient(hunting): picked mon.noname-c con 0x7fb1b00b4820 addr v1:172.21.15.71:6790/0 2024-03-29T11:03:53.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.779+0000 7fb1af59e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1b0071070 con 0x7fb1b00b3cd0 2024-03-29T11:03:53.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.779+0000 7fb1af59e640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1b00ab5c0 con 0x7fb1b00a9ce0 2024-03-29T11:03:53.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.779+0000 7fb1af59e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1b0143eb0 con 0x7fb1b00b4820 2024-03-29T11:03:53.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.779+0000 7fb1af59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:53.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.779+0000 7fb1af59e640 10 monclient(hunting): authenticate will time out at 990.037780s 2024-03-29T11:03:53.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.779+0000 7fb1aed9d640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fb1b00b4820 0x7fb1b01de910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:48182/0 (socket says 172.21.15.71:48182) 2024-03-29T11:03:53.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.779+0000 7fb1aed9d640 1 -- 172.21.15.71:0/1210391663 learned_addr learned my addr 172.21.15.71:0/1210391663 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:03:53.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.780+0000 7fb19effd640 10 client.?.objecter ms_handle_connect 0x7fb1b00b3cd0 2024-03-29T11:03:53.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.780+0000 7fb19effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:53.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.780+0000 7fb19effd640 10 client.?.objecter ms_handle_connect 0x7fb1b00a9ce0 2024-03-29T11:03:53.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.780+0000 7fb19effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:53.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.780+0000 7fb19effd640 10 client.?.objecter ms_handle_connect 0x7fb1b00b4820 2024-03-29T11:03:53.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.780+0000 7fb19effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:53.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 1 -- 172.21.15.71:0/1210391663 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098228511 0 0) 0x7fb1b00ab5c0 con 0x7fb1b00a9ce0 2024-03-29T11:03:53.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 10 monclient(hunting): _negotiate 2024-03-29T11:03:53.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:53.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:53.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 10 monclient(hunting): my global_id is 4324 2024-03-29T11:03:53.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 1 -- 172.21.15.71:0/1210391663 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb190004eb0 con 0x7fb1b00a9ce0 2024-03-29T11:03:53.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 1 -- 172.21.15.71:0/1210391663 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652824451 0 0) 0x7fb1b0071070 con 0x7fb1b00b3cd0 2024-03-29T11:03:53.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 10 monclient(hunting): _negotiate 2024-03-29T11:03:53.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:53.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:53.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:03:53.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 1 -- 172.21.15.71:0/1210391663 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1b00ab5c0 con 0x7fb1b00b3cd0 2024-03-29T11:03:53.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 1 -- 172.21.15.71:0/1210391663 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730674223 0 0) 0x7fb1b0143eb0 con 0x7fb1b00b4820 2024-03-29T11:03:53.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 10 monclient(hunting): _negotiate 2024-03-29T11:03:53.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:53.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:53.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 10 monclient(hunting): my global_id is 4367 2024-03-29T11:03:53.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 1 -- 172.21.15.71:0/1210391663 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1b0071070 con 0x7fb1b00b4820 2024-03-29T11:03:53.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 1 -- 172.21.15.71:0/1210391663 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2455987082 0 0) 0x7fb190004eb0 con 0x7fb1b00a9ce0 2024-03-29T11:03:53.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 1 -- 172.21.15.71:0/1210391663 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1b0143eb0 con 0x7fb1b00a9ce0 2024-03-29T11:03:53.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 1 -- 172.21.15.71:0/1210391663 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb1a0001780 con 0x7fb1b00a9ce0 2024-03-29T11:03:53.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:53.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:03:53.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 10 monclient(hunting): dump: 2024-03-29T11:03:53.767 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:53.767 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:53.767 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:53.767 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:53.767 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:53.767 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:53.767 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:53.767 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:53.768 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:53.768 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:53.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:03:53.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 1 -- 172.21.15.71:0/1210391663 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 839334880 0 0) 0x7fb1b00ab5c0 con 0x7fb1b00b3cd0 2024-03-29T11:03:53.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 1 -- 172.21.15.71:0/1210391663 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb190004eb0 con 0x7fb1b00b3cd0 2024-03-29T11:03:53.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 1 -- 172.21.15.71:0/1210391663 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb1a8001d30 con 0x7fb1b00b3cd0 2024-03-29T11:03:53.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:53.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:03:53.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.781+0000 7fb19effd640 10 monclient(hunting): dump: 2024-03-29T11:03:53.768 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:53.768 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:53.768 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:53.768 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:53.768 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:53.768 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:53.769 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:53.769 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:53.769 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:53.769 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:53.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.782+0000 7fb19effd640 1 -- 172.21.15.71:0/1210391663 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4276525286 0 0) 0x7fb1b0071070 con 0x7fb1b00b4820 2024-03-29T11:03:53.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.782+0000 7fb19effd640 1 -- 172.21.15.71:0/1210391663 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1b00ab5c0 con 0x7fb1b00b4820 2024-03-29T11:03:53.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.782+0000 7fb19effd640 1 -- 172.21.15.71:0/1210391663 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb1a4001930 con 0x7fb1b00b4820 2024-03-29T11:03:53.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.782+0000 7fb19effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:53.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.782+0000 7fb19effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:53.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.782+0000 7fb19effd640 10 monclient(hunting): dump: 2024-03-29T11:03:53.769 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:53.769 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:53.769 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:53.769 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:53.769 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:53.770 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:53.770 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:53.770 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:53.770 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:53.770 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:53.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.782+0000 7fb19effd640 1 -- 172.21.15.71:0/1210391663 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2343590088 0 0) 0x7fb1b0143eb0 con 0x7fb1b00a9ce0 2024-03-29T11:03:53.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.782+0000 7fb19effd640 1 -- 172.21.15.71:0/1210391663 >> v1:172.21.15.71:6790/0 conn(0x7fb1b00b4820 legacy=0x7fb1b01de910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:53.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.782+0000 7fb19effd640 1 -- 172.21.15.71:0/1210391663 >> v1:172.21.15.71:6789/0 conn(0x7fb1b00b3cd0 legacy=0x7fb1b0139cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:53.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.782+0000 7fb19effd640 10 monclient: _finish_hunting 0 2024-03-29T11:03:53.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.782+0000 7fb19effd640 1 monclient: found mon.b 2024-03-29T11:03:53.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.782+0000 7fb19effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:53.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.782+0000 7fb19effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:03:53.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.782+0000 7fb19effd640 1 -- 172.21.15.71:0/1210391663 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b013a940 con 0x7fb1b00a9ce0 2024-03-29T11:03:53.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.782+0000 7fb19effd640 10 monclient: _finish_auth 0 2024-03-29T11:03:53.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.782+0000 7fb19effd640 10 monclient: _check_auth_tickets 2024-03-29T11:03:53.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.782+0000 7fb19effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:53.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.782+0000 7fb1af59e640 5 monclient: authenticate success, global_id 4324 2024-03-29T11:03:53.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.783+0000 7fb1af59e640 10 monclient: _renew_subs 2024-03-29T11:03:53.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.783+0000 7fb1af59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:03:53.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.783+0000 7fb1af59e640 1 -- 172.21.15.71:0/1210391663 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1b013a400 con 0x7fb1b00a9ce0 2024-03-29T11:03:53.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.783+0000 7fb1af59e640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:03:53.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.783+0000 7fb1af59e640 10 monclient: _renew_subs 2024-03-29T11:03:53.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.783+0000 7fb1af59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:03:53.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.783+0000 7fb1af59e640 1 -- 172.21.15.71:0/1210391663 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1b01d7140 con 0x7fb1b00a9ce0 2024-03-29T11:03:53.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.783+0000 7fb1af59e640 1 librados: init done 2024-03-29T11:03:53.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.783+0000 7fb19effd640 1 -- 172.21.15.71:0/1210391663 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1a0001d50 con 0x7fb1b00a9ce0 2024-03-29T11:03:53.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.783+0000 7fb19effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:53.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.783+0000 7fb19effd640 1 -- 172.21.15.71:0/1210391663 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb1a00037b0 con 0x7fb1b00a9ce0 2024-03-29T11:03:53.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.783+0000 7fb19effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:53.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.783+0000 7fb19effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:03:53.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.783+0000 7fb19effd640 10 monclient: dump: 2024-03-29T11:03:53.773 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:03:53.773 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:03:53.773 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:53.773 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:03:53.773 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:53.774 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:03:53.774 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:03:53.774 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:03:53.774 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:03:53.774 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:03:53.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.783+0000 7fb1acd99640 4 set_mon_vals no callback set 2024-03-29T11:03:53.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.784+0000 7fb1af59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:53.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.784+0000 7fb1af59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:53.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.784+0000 7fb1af59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:03:53.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.784+0000 7fb1af59e640 1 -- 172.21.15.71:0/1210391663 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb194005080 con 0x7fb1b00a9ce0 2024-03-29T11:03:53.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.785+0000 7fb19effd640 1 -- 172.21.15.71:0/1210391663 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7fb1a0019780 con 0x7fb1b00a9ce0 2024-03-29T11:03:53.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.786+0000 7fb1add9b640 10 monclient: get_auth_request con 0x7fb19006e0e0 auth_method 0 2024-03-29T11:03:53.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.786+0000 7fb19effd640 1 -- 172.21.15.71:0/1210391663 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7fb1a0083740 con 0x7fb1b00a9ce0 2024-03-29T11:03:53.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.786+0000 7fb19effd640 10 client.4324.objecter ms_dispatch 0x7fb1b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:03:53.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.786+0000 7fb19effd640 3 client.4324.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:03:53.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.786+0000 7fb19effd640 3 client.4324.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:03:53.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.786+0000 7fb19effd640 20 client.4324.objecter dump_active .. 0 homeless 2024-03-29T11:03:53.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.786+0000 7fb19effd640 10 client.4324.objecter ms_handle_connect 0x7fb19006e0e0 2024-03-29T11:03:53.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.792+0000 7fb19effd640 1 -- 172.21.15.71:0/1210391663 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb1a004f020 con 0x7fb1b00a9ce0 2024-03-29T11:03:53.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.792+0000 7fb19effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:53.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:53.792+0000 7fb19effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:03:54.163 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:54.183+0000 7fb1af59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:54.163 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:54.183+0000 7fb1af59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:54.163 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:54.183+0000 7fb1af59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:03:54.163 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:54.183+0000 7fb1af59e640 1 -- 172.21.15.71:0/1210391663 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb194005330 con 0x7fb1b00a9ce0 2024-03-29T11:03:54.164 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:54.184+0000 7fb19effd640 1 -- 172.21.15.71:0/1210391663 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7fb1a0050f50 con 0x7fb1b00a9ce0 2024-03-29T11:03:54.164 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:54.184+0000 7fb19effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:54.164 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:54.184+0000 7fb19effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:03:54.164 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:03:54.164 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:03:54.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:54.188+0000 7fb1af59e640 10 librados: watch_flush enter 2024-03-29T11:03:54.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:54.188+0000 7fb1af59e640 10 librados: watch_flush exit 2024-03-29T11:03:54.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:54.188+0000 7fb1af59e640 20 client.4324.objecter shutdown clearing up homeless session... 2024-03-29T11:03:54.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:54.188+0000 7fb1af59e640 10 client.4324.objecter successfully canceled tick 2024-03-29T11:03:54.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:54.188+0000 7fb1af59e640 1 -- 172.21.15.71:0/1210391663 >> v1:172.21.15.137:6816/973332665 conn(0x7fb19006e0e0 legacy=0x7fb1900705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:54.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:54.188+0000 7fb1af59e640 10 monclient: shutdown 2024-03-29T11:03:54.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:54.188+0000 7fb1af59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:54.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:54.188+0000 7fb1af59e640 1 -- 172.21.15.71:0/1210391663 >> v1:172.21.15.137:6789/0 conn(0x7fb1b00a9ce0 legacy=0x7fb1b01437a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:54.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:54.188+0000 7fb1af59e640 1 -- 172.21.15.71:0/1210391663 shutdown_connections 2024-03-29T11:03:54.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:54.188+0000 7fb1af59e640 1 -- 172.21.15.71:0/1210391663 >> 172.21.15.71:0/1210391663 conn(0x7fb1b006d070 msgr2=0x7fb1b0071880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:54.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:54.189+0000 7fb1af59e640 1 -- 172.21.15.71:0/1210391663 shutdown_connections 2024-03-29T11:03:54.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:54.189+0000 7fb1af59e640 1 -- 172.21.15.71:0/1210391663 wait complete. 2024-03-29T11:03:54.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:03:54.189+0000 7fb1af59e640 1 librados: shutdown 2024-03-29T11:03:54.184 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:04:00.990 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:04:01.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.152+0000 7f73b2c4e640 10 monclient: get_monmap_and_config 2024-03-29T11:04:01.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.152+0000 7f73b2c4e640 10 monclient: build_initial_monmap 2024-03-29T11:04:01.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.152+0000 7f73b2c4e640 10 monclient: monmap: 2024-03-29T11:04:01.132 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:04:01.132 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:01.132 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:04:01.153336+0000 2024-03-29T11:04:01.132 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:04:01.153336+0000 2024-03-29T11:04:01.132 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:01.132 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:01.132 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:04:01.132 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:04:01.132 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:04:01.132 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:01.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.153+0000 7f73b2c4e640 1 Processor -- start 2024-03-29T11:04:01.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.153+0000 7f73b2c4e640 1 -- start start 2024-03-29T11:04:01.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.153+0000 7f73b2c4e640 10 monclient: init 2024-03-29T11:04:01.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.154+0000 7f73b2c4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:01.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.154+0000 7f73b2c4e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:04:01.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.154+0000 7f73b2c4e640 10 monclient(hunting): picked mon.noname-c con 0x7f73ac1457f0 addr v1:172.21.15.137:6789/0 2024-03-29T11:04:01.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.154+0000 7f73b2c4e640 10 monclient(hunting): picked mon.noname-b con 0x7f73ac13f650 addr v1:172.21.15.71:6790/0 2024-03-29T11:04:01.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.154+0000 7f73b2c4e640 10 monclient(hunting): picked mon.noname-a con 0x7f73ac0a9d70 addr v1:172.21.15.71:6789/0 2024-03-29T11:04:01.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.154+0000 7f73b2c4e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ac0aaef0 con 0x7f73ac0a9d70 2024-03-29T11:04:01.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.154+0000 7f73b2c4e640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ac145bd0 con 0x7f73ac1457f0 2024-03-29T11:04:01.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.154+0000 7f73b2c4e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ac145eb0 con 0x7f73ac13f650 2024-03-29T11:04:01.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.154+0000 7f73b2c4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:01.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.154+0000 7f73b2c4e640 10 monclient(hunting): authenticate will time out at 997.412831s 2024-03-29T11:04:01.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.154+0000 7f73abfff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f73ac13f650 0x7f73ac0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:34348/0 (socket says 172.21.15.71:34348) 2024-03-29T11:04:01.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.154+0000 7f73abfff640 1 -- 172.21.15.71:0/3179725296 learned_addr learned my addr 172.21.15.71:0/3179725296 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:04:01.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.155+0000 7f73ab7fe640 1 -- 172.21.15.71:0/3179725296 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4151246574 0 0) 0x7f73ac145eb0 con 0x7f73ac13f650 2024-03-29T11:04:01.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.155+0000 7f73ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:04:01.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.155+0000 7f73ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:01.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.155+0000 7f73ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:01.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.155+0000 7f73ab7fe640 10 monclient(hunting): my global_id is 4370 2024-03-29T11:04:01.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.155+0000 7f73ab7fe640 1 -- 172.21.15.71:0/3179725296 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f739c003520 con 0x7f73ac13f650 2024-03-29T11:04:01.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.155+0000 7f73ab7fe640 1 -- 172.21.15.71:0/3179725296 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2811031472 0 0) 0x7f739c003520 con 0x7f73ac13f650 2024-03-29T11:04:01.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.155+0000 7f73ab7fe640 1 -- 172.21.15.71:0/3179725296 >> v1:172.21.15.137:6789/0 conn(0x7f73ac1457f0 legacy=0x7f73ac13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:04:01.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.155+0000 7f73ab7fe640 1 -- 172.21.15.71:0/3179725296 >> v1:172.21.15.71:6789/0 conn(0x7f73ac0a9d70 legacy=0x7f73ac14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:01.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.155+0000 7f73ab7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:04:01.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.155+0000 7f73ab7fe640 1 monclient: found mon.noname-b 2024-03-29T11:04:01.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.155+0000 7f73ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:01.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.155+0000 7f73ab7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:04:01.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.155+0000 7f73ab7fe640 1 -- 172.21.15.71:0/3179725296 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ac146340 con 0x7f73ac13f650 2024-03-29T11:04:01.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.155+0000 7f73ab7fe640 10 monclient: _finish_auth 0 2024-03-29T11:04:01.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.155+0000 7f73ab7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:04:01.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.155+0000 7f73ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:01.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.155+0000 7f73ab7fe640 1 -- 172.21.15.71:0/3179725296 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f7398001690 con 0x7f73ac13f650 2024-03-29T11:04:01.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.155+0000 7f73ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:01.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.155+0000 7f73ab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:04:01.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.155+0000 7f73ab7fe640 10 monclient: dump: 2024-03-29T11:04:01.137 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:01.137 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:01.137 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:01.137 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:01.137 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:01.137 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:01.137 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:01.137 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:01.137 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:01.138 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:01.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.155+0000 7f73b2c4e640 5 monclient: authenticate success, global_id 4370 2024-03-29T11:04:01.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.155+0000 7f73b2c4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:04:01.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.156+0000 7f73ab7fe640 1 -- 172.21.15.71:0/3179725296 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7398001920 con 0x7f73ac13f650 2024-03-29T11:04:01.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.156+0000 7f73ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:01.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.156+0000 7f73ab7fe640 1 -- 172.21.15.71:0/3179725296 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f7398001cb0 con 0x7f73ac13f650 2024-03-29T11:04:01.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.156+0000 7f73ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:01.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.156+0000 7f73ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:04:01.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.156+0000 7f73ab7fe640 10 monclient: dump: 2024-03-29T11:04:01.138 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:01.138 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:01.138 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:01.138 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:01.138 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:01.139 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:01.139 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:01.139 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:01.139 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:01.139 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:01.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.156+0000 7f73b2c4e640 10 monclient: get_monmap_and_config success 2024-03-29T11:04:01.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.156+0000 7f73b2c4e640 4 set_mon_vals no callback set 2024-03-29T11:04:01.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.157+0000 7f73b2c4e640 10 monclient: shutdown 2024-03-29T11:04:01.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.157+0000 7f73b2c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:01.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.157+0000 7f73b2c4e640 1 -- 172.21.15.71:0/3179725296 >> v1:172.21.15.71:6790/0 conn(0x7f73ac13f650 legacy=0x7f73ac0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:01.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.157+0000 7f73b2c4e640 1 -- 172.21.15.71:0/3179725296 shutdown_connections 2024-03-29T11:04:01.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.157+0000 7f73b2c4e640 1 -- 172.21.15.71:0/3179725296 >> 172.21.15.71:0/3179725296 conn(0x7f73ac066a00 msgr2=0x7f73ac066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:01.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.157+0000 7f73b2c4e640 1 -- 172.21.15.71:0/3179725296 shutdown_connections 2024-03-29T11:04:01.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.157+0000 7f73b2c4e640 1 -- 172.21.15.71:0/3179725296 wait complete. 2024-03-29T11:04:01.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.158+0000 7f73b2c4e640 10 monclient: build_initial_monmap 2024-03-29T11:04:01.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.158+0000 7f73b2c4e640 10 monclient: monmap: 2024-03-29T11:04:01.141 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:04:01.141 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:04:01.141 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:04:01.142 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:04:01.142 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:01.142 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:01.142 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:04:01.142 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:04:01.142 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:04:01.142 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:01.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.159+0000 7f73b2c4e640 1 librados: starting msgr at 2024-03-29T11:04:01.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.159+0000 7f73b2c4e640 1 librados: starting objecter 2024-03-29T11:04:01.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.159+0000 7f73b2c4e640 1 Processor -- start 2024-03-29T11:04:01.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.159+0000 7f73b2c4e640 1 -- start start 2024-03-29T11:04:01.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.159+0000 7f73b2c4e640 1 librados: setting wanted keys 2024-03-29T11:04:01.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.159+0000 7f73b2c4e640 1 librados: calling monclient init 2024-03-29T11:04:01.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.159+0000 7f73b2c4e640 10 monclient: init 2024-03-29T11:04:01.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.160+0000 7f73b2c4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:01.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.160+0000 7f73b2c4e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:04:01.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.160+0000 7f73b2c4e640 10 monclient(hunting): picked mon.noname-a con 0x7f73ac0a9d70 addr v1:172.21.15.71:6789/0 2024-03-29T11:04:01.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.160+0000 7f73b09ea640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f73ac0a9d70 0x7f73ac0b06d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:51286/0 (socket says 172.21.15.71:51286) 2024-03-29T11:04:01.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.160+0000 7f73b09ea640 1 -- 172.21.15.71:0/3465990463 learned_addr learned my addr 172.21.15.71:0/3465990463 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:04:01.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.160+0000 7f73b2c4e640 10 monclient(hunting): picked mon.noname-c con 0x7f73ac13f650 addr v1:172.21.15.71:6790/0 2024-03-29T11:04:01.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.161+0000 7f73b2c4e640 10 monclient(hunting): picked mon.noname-b con 0x7f73ac1457f0 addr v1:172.21.15.137:6789/0 2024-03-29T11:04:01.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.161+0000 7f73a97fa640 10 client.?.objecter ms_handle_connect 0x7f73ac0a9d70 2024-03-29T11:04:01.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.161+0000 7f73b2c4e640 1 -- 172.21.15.71:0/3465990463 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ac0aeee0 con 0x7f73ac0a9d70 2024-03-29T11:04:01.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.161+0000 7f73a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:01.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.161+0000 7f73b2c4e640 1 -- 172.21.15.71:0/3465990463 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ac0affd0 con 0x7f73ac1457f0 2024-03-29T11:04:01.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.161+0000 7f73a97fa640 10 client.?.objecter ms_handle_connect 0x7f73ac13f650 2024-03-29T11:04:01.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.161+0000 7f73a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:01.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.161+0000 7f73b2c4e640 1 -- 172.21.15.71:0/3465990463 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ac0b02b0 con 0x7f73ac13f650 2024-03-29T11:04:01.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.161+0000 7f73b2c4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:01.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.161+0000 7f73b2c4e640 10 monclient(hunting): authenticate will time out at 997.419675s 2024-03-29T11:04:01.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.161+0000 7f73a97fa640 1 -- 172.21.15.71:0/3465990463 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2087240633 0 0) 0x7f73ac0aeee0 con 0x7f73ac0a9d70 2024-03-29T11:04:01.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.161+0000 7f73a97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:04:01.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.161+0000 7f73a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:01.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.161+0000 7f73a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:01.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.161+0000 7f73a97fa640 10 monclient(hunting): my global_id is 4374 2024-03-29T11:04:01.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.161+0000 7f73a97fa640 1 -- 172.21.15.71:0/3465990463 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7390004eb0 con 0x7f73ac0a9d70 2024-03-29T11:04:01.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.162+0000 7f73a97fa640 1 -- 172.21.15.71:0/3465990463 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2961322322 0 0) 0x7f7390004eb0 con 0x7f73ac0a9d70 2024-03-29T11:04:01.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.162+0000 7f73a97fa640 1 -- 172.21.15.71:0/3465990463 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73ac0aeee0 con 0x7f73ac0a9d70 2024-03-29T11:04:01.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.162+0000 7f73a97fa640 1 -- 172.21.15.71:0/3465990463 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f73a00017d0 con 0x7f73ac0a9d70 2024-03-29T11:04:01.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.162+0000 7f73a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:01.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.162+0000 7f73a97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:04:01.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.162+0000 7f73a97fa640 10 monclient(hunting): dump: 2024-03-29T11:04:01.149 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:01.149 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:01.149 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:01.149 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:01.149 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:01.149 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:01.149 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:01.149 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:01.149 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:01.149 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:01.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.162+0000 7f73a97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:04:01.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.162+0000 7f73a97fa640 1 -- 172.21.15.71:0/3465990463 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 207459407 0 0) 0x7f73ac0b02b0 con 0x7f73ac13f650 2024-03-29T11:04:01.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.162+0000 7f73a97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:04:01.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.162+0000 7f73a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:01.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.162+0000 7f73a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:01.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.162+0000 7f73a97fa640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:04:01.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.162+0000 7f73a97fa640 1 -- 172.21.15.71:0/3465990463 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7390004eb0 con 0x7f73ac13f650 2024-03-29T11:04:01.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.162+0000 7f73a97fa640 1 -- 172.21.15.71:0/3465990463 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2084768269 0 0) 0x7f73ac0aeee0 con 0x7f73ac0a9d70 2024-03-29T11:04:01.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.162+0000 7f73a97fa640 1 -- 172.21.15.71:0/3465990463 >> v1:172.21.15.71:6790/0 conn(0x7f73ac13f650 legacy=0x7f73ac0b0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:01.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.162+0000 7f73a97fa640 1 -- 172.21.15.71:0/3465990463 >> v1:172.21.15.137:6789/0 conn(0x7f73ac1457f0 legacy=0x7f73ac1e19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:01.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.162+0000 7f73a97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:04:01.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.162+0000 7f73a97fa640 1 monclient: found mon.a 2024-03-29T11:04:01.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.162+0000 7f73a97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:01.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.162+0000 7f73a97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:04:01.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.162+0000 7f73a97fa640 1 -- 172.21.15.71:0/3465990463 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ac0aea40 con 0x7f73ac0a9d70 2024-03-29T11:04:01.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.162+0000 7f73a97fa640 10 monclient: _finish_auth 0 2024-03-29T11:04:01.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.162+0000 7f73a97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:04:01.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.162+0000 7f73a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:01.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.163+0000 7f73b2c4e640 5 monclient: authenticate success, global_id 4374 2024-03-29T11:04:01.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.163+0000 7f73b2c4e640 10 monclient: _renew_subs 2024-03-29T11:04:01.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.163+0000 7f73b2c4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:04:01.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.163+0000 7f73b2c4e640 1 -- 172.21.15.71:0/3465990463 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73ac0b1680 con 0x7f73ac0a9d70 2024-03-29T11:04:01.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.163+0000 7f73b2c4e640 10 client.4374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:04:01.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.163+0000 7f73b2c4e640 10 monclient: _renew_subs 2024-03-29T11:04:01.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.163+0000 7f73b2c4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:04:01.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.163+0000 7f73b2c4e640 1 -- 172.21.15.71:0/3465990463 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f73ac0ae030 con 0x7f73ac0a9d70 2024-03-29T11:04:01.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.163+0000 7f73b2c4e640 1 librados: init done 2024-03-29T11:04:01.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.163+0000 7f73a97fa640 1 -- 172.21.15.71:0/3465990463 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73a0001930 con 0x7f73ac0a9d70 2024-03-29T11:04:01.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.163+0000 7f73a97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:01.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.163+0000 7f73a97fa640 1 -- 172.21.15.71:0/3465990463 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f73a0004a00 con 0x7f73ac0a9d70 2024-03-29T11:04:01.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.163+0000 7f73aa7fc640 4 set_mon_vals no callback set 2024-03-29T11:04:01.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.163+0000 7f73a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:01.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.163+0000 7f73a97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:04:01.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.163+0000 7f73a97fa640 10 monclient: dump: 2024-03-29T11:04:01.152 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:01.152 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:01.152 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:01.152 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:01.152 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:01.152 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:01.152 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:01.152 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:01.152 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:01.153 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:01.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.166+0000 7f73a97fa640 1 -- 172.21.15.71:0/3465990463 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f73a001a9d0 con 0x7f73ac0a9d70 2024-03-29T11:04:01.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.166+0000 7f73b2c4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:01.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.166+0000 7f73b2c4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:01.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.166+0000 7f73b2c4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:04:01.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.166+0000 7f73a97fa640 1 -- 172.21.15.71:0/3465990463 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f73a0084a20 con 0x7f73ac0a9d70 2024-03-29T11:04:01.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.166+0000 7f73b2c4e640 1 -- 172.21.15.71:0/3465990463 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f738c005080 con 0x7f73ac0a9d70 2024-03-29T11:04:01.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.166+0000 7f73a97fa640 10 client.4374.objecter ms_dispatch 0x7f73ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:04:01.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.166+0000 7f73a97fa640 3 client.4374.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:04:01.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.166+0000 7f73a97fa640 3 client.4374.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:04:01.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.167+0000 7f73a97fa640 20 client.4374.objecter dump_active .. 0 homeless 2024-03-29T11:04:01.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.167+0000 7f73abfff640 10 monclient: get_auth_request con 0x7f739006dd60 auth_method 0 2024-03-29T11:04:01.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.168+0000 7f73a97fa640 10 client.4374.objecter ms_handle_connect 0x7f739006dd60 2024-03-29T11:04:01.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.176+0000 7f73a97fa640 1 -- 172.21.15.71:0/3465990463 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f73a0050130 con 0x7f73ac0a9d70 2024-03-29T11:04:01.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.176+0000 7f73a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:01.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.176+0000 7f73a97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:04:01.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.533+0000 7f73b2c4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:01.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.533+0000 7f73b2c4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:01.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.533+0000 7f73b2c4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:04:01.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.533+0000 7f73b2c4e640 1 -- 172.21.15.71:0/3465990463 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f738c005330 con 0x7f73ac0a9d70 2024-03-29T11:04:01.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.535+0000 7f73a97fa640 1 -- 172.21.15.71:0/3465990463 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f73a0052230 con 0x7f73ac0a9d70 2024-03-29T11:04:01.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.535+0000 7f73a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:01.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.535+0000 7f73a97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:04:01.514 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:04:01.514 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:04:01.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.538+0000 7f73b2c4e640 10 librados: watch_flush enter 2024-03-29T11:04:01.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.538+0000 7f73b2c4e640 10 librados: watch_flush exit 2024-03-29T11:04:01.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.538+0000 7f73b2c4e640 20 client.4374.objecter shutdown clearing up homeless session... 2024-03-29T11:04:01.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.538+0000 7f73b2c4e640 10 client.4374.objecter successfully canceled tick 2024-03-29T11:04:01.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.538+0000 7f73b2c4e640 1 -- 172.21.15.71:0/3465990463 >> v1:172.21.15.137:6816/973332665 conn(0x7f739006dd60 legacy=0x7f7390070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:01.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.538+0000 7f73b2c4e640 10 monclient: shutdown 2024-03-29T11:04:01.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.538+0000 7f73b2c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:01.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.538+0000 7f73b2c4e640 1 -- 172.21.15.71:0/3465990463 >> v1:172.21.15.71:6789/0 conn(0x7f73ac0a9d70 legacy=0x7f73ac0b06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:01.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.539+0000 7f73b2c4e640 1 -- 172.21.15.71:0/3465990463 shutdown_connections 2024-03-29T11:04:01.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.539+0000 7f73b2c4e640 1 -- 172.21.15.71:0/3465990463 >> 172.21.15.71:0/3465990463 conn(0x7f73ac066a00 msgr2=0x7f73ac0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:01.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.539+0000 7f73b2c4e640 1 -- 172.21.15.71:0/3465990463 shutdown_connections 2024-03-29T11:04:01.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.539+0000 7f73b2c4e640 1 -- 172.21.15.71:0/3465990463 wait complete. 2024-03-29T11:04:01.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:01.539+0000 7f73b2c4e640 1 librados: shutdown 2024-03-29T11:04:01.533 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:04:08.341 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:04:08.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.506+0000 7f7babbba640 10 monclient: get_monmap_and_config 2024-03-29T11:04:08.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.506+0000 7f7babbba640 10 monclient: build_initial_monmap 2024-03-29T11:04:08.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.506+0000 7f7babbba640 10 monclient: monmap: 2024-03-29T11:04:08.535 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:04:08.536 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:08.536 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:04:08.507298+0000 2024-03-29T11:04:08.536 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:04:08.507298+0000 2024-03-29T11:04:08.536 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:08.536 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:08.536 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:04:08.536 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:04:08.536 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:04:08.536 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:08.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.507+0000 7f7babbba640 1 Processor -- start 2024-03-29T11:04:08.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.507+0000 7f7babbba640 1 -- start start 2024-03-29T11:04:08.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.507+0000 7f7babbba640 10 monclient: init 2024-03-29T11:04:08.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.508+0000 7f7babbba640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:08.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.508+0000 7f7babbba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:04:08.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.508+0000 7f7babbba640 10 monclient(hunting): picked mon.noname-a con 0x7f7ba40ad6c0 addr v1:172.21.15.71:6789/0 2024-03-29T11:04:08.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.508+0000 7f7babbba640 10 monclient(hunting): picked mon.noname-c con 0x7f7ba40af6c0 addr v1:172.21.15.137:6789/0 2024-03-29T11:04:08.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.508+0000 7f7babbba640 10 monclient(hunting): picked mon.noname-b con 0x7f7ba414fdc0 addr v1:172.21.15.71:6790/0 2024-03-29T11:04:08.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.508+0000 7f7babbba640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba40a94e0 con 0x7f7ba40ad6c0 2024-03-29T11:04:08.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.508+0000 7f7babbba640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba40af3a0 con 0x7f7ba40af6c0 2024-03-29T11:04:08.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.508+0000 7f7babbba640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba40afb60 con 0x7f7ba414fdc0 2024-03-29T11:04:08.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.508+0000 7f7babbba640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:08.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.508+0000 7f7babbba640 10 monclient(hunting): authenticate will time out at 1004.766491s 2024-03-29T11:04:08.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.509+0000 7f7ba9155640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f7ba40af6c0 0x7f7ba4152980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:56394/0 (socket says 172.21.15.71:56394) 2024-03-29T11:04:08.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.509+0000 7f7ba9155640 1 -- 172.21.15.71:0/3158116331 learned_addr learned my addr 172.21.15.71:0/3158116331 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:04:08.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.510+0000 7f7ba8954640 1 -- 172.21.15.71:0/3158116331 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 438411704 0 0) 0x7f7ba40af3a0 con 0x7f7ba40af6c0 2024-03-29T11:04:08.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.510+0000 7f7ba8954640 10 monclient(hunting): _negotiate 2024-03-29T11:04:08.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.510+0000 7f7ba8954640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:08.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.510+0000 7f7ba8954640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:08.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.510+0000 7f7ba8954640 10 monclient(hunting): my global_id is 4327 2024-03-29T11:04:08.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.510+0000 7f7ba8954640 1 -- 172.21.15.71:0/3158116331 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b90003520 con 0x7f7ba40af6c0 2024-03-29T11:04:08.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.512+0000 7f7ba8954640 1 -- 172.21.15.71:0/3158116331 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1177666066 0 0) 0x7f7b90003520 con 0x7f7ba40af6c0 2024-03-29T11:04:08.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.554+0000 7f7ba8954640 1 -- 172.21.15.71:0/3158116331 >> v1:172.21.15.71:6790/0 conn(0x7f7ba414fdc0 legacy=0x7f7ba415a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:04:08.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.554+0000 7f7ba8954640 1 -- 172.21.15.71:0/3158116331 >> v1:172.21.15.71:6789/0 conn(0x7f7ba40ad6c0 legacy=0x7f7ba414f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:04:08.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.554+0000 7f7ba8954640 10 monclient: _finish_hunting 0 2024-03-29T11:04:08.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.554+0000 7f7ba8954640 1 monclient: found mon.noname-c 2024-03-29T11:04:08.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.554+0000 7f7ba8954640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:08.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.554+0000 7f7ba8954640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.137:6789/0 2024-03-29T11:04:08.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.554+0000 7f7ba8954640 1 -- 172.21.15.71:0/3158116331 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba40aab80 con 0x7f7ba40af6c0 2024-03-29T11:04:08.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.554+0000 7f7ba8954640 10 monclient: _finish_auth 0 2024-03-29T11:04:08.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.554+0000 7f7ba8954640 10 monclient: _check_auth_tickets 2024-03-29T11:04:08.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.554+0000 7f7ba8954640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:08.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.554+0000 7f7ba8954640 1 -- 172.21.15.71:0/3158116331 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f7b94001690 con 0x7f7ba40af6c0 2024-03-29T11:04:08.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.554+0000 7f7ba8954640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:08.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.554+0000 7f7ba8954640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:04:08.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.554+0000 7f7ba8954640 10 monclient: dump: 2024-03-29T11:04:08.539 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:08.539 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:08.539 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:08.539 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:08.539 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:08.539 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:08.539 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:08.539 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:08.539 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:08.539 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:08.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.555+0000 7f7ba8954640 1 -- 172.21.15.71:0/3158116331 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b94001860 con 0x7f7ba40af6c0 2024-03-29T11:04:08.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.555+0000 7f7ba8954640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:08.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.555+0000 7f7ba8954640 1 -- 172.21.15.71:0/3158116331 <== mon.1 v1:172.21.15.137:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f7b94001cb0 con 0x7f7ba40af6c0 2024-03-29T11:04:08.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.555+0000 7f7ba8954640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:08.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.555+0000 7f7ba8954640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:04:08.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.555+0000 7f7ba8954640 10 monclient: dump: 2024-03-29T11:04:08.540 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:08.540 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:08.540 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:08.540 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:08.540 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:08.540 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:08.540 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:08.540 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:08.540 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:08.541 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:08.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.556+0000 7f7babbba640 5 monclient: authenticate success, global_id 4327 2024-03-29T11:04:08.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.556+0000 7f7babbba640 10 monclient: get_monmap_and_config success 2024-03-29T11:04:08.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.556+0000 7f7babbba640 4 set_mon_vals no callback set 2024-03-29T11:04:08.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.557+0000 7f7babbba640 10 monclient: shutdown 2024-03-29T11:04:08.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.557+0000 7f7babbba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:08.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.557+0000 7f7babbba640 1 -- 172.21.15.71:0/3158116331 >> v1:172.21.15.137:6789/0 conn(0x7f7ba40af6c0 legacy=0x7f7ba4152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:08.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.557+0000 7f7babbba640 1 -- 172.21.15.71:0/3158116331 shutdown_connections 2024-03-29T11:04:08.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.558+0000 7f7babbba640 1 -- 172.21.15.71:0/3158116331 >> 172.21.15.71:0/3158116331 conn(0x7f7ba40b35e0 msgr2=0x7f7ba40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:08.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.559+0000 7f7babbba640 1 -- 172.21.15.71:0/3158116331 shutdown_connections 2024-03-29T11:04:08.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.559+0000 7f7babbba640 1 -- 172.21.15.71:0/3158116331 wait complete. 2024-03-29T11:04:08.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.560+0000 7f7babbba640 10 monclient: build_initial_monmap 2024-03-29T11:04:08.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.561+0000 7f7babbba640 10 monclient: monmap: 2024-03-29T11:04:08.544 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:04:08.544 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:04:08.544 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:04:08.544 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:04:08.544 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:08.544 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:08.545 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:04:08.545 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:04:08.545 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:04:08.545 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:08.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.561+0000 7f7babbba640 1 librados: starting msgr at 2024-03-29T11:04:08.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.561+0000 7f7babbba640 1 librados: starting objecter 2024-03-29T11:04:08.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.561+0000 7f7babbba640 1 Processor -- start 2024-03-29T11:04:08.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.561+0000 7f7babbba640 1 -- start start 2024-03-29T11:04:08.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.561+0000 7f7babbba640 1 librados: setting wanted keys 2024-03-29T11:04:08.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.561+0000 7f7babbba640 1 librados: calling monclient init 2024-03-29T11:04:08.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.561+0000 7f7babbba640 10 monclient: init 2024-03-29T11:04:08.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.562+0000 7f7babbba640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:08.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.562+0000 7f7babbba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:04:08.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.562+0000 7f7babbba640 10 monclient(hunting): picked mon.noname-a con 0x7f7ba40ad6c0 addr v1:172.21.15.71:6789/0 2024-03-29T11:04:08.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.562+0000 7f7babbba640 10 monclient(hunting): picked mon.noname-c con 0x7f7ba40af6c0 addr v1:172.21.15.71:6790/0 2024-03-29T11:04:08.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.562+0000 7f7babbba640 10 monclient(hunting): picked mon.noname-b con 0x7f7ba414fdc0 addr v1:172.21.15.137:6789/0 2024-03-29T11:04:08.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.562+0000 7f7babbba640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba408a350 con 0x7f7ba40ad6c0 2024-03-29T11:04:08.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.562+0000 7f7babbba640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba408a630 con 0x7f7ba414fdc0 2024-03-29T11:04:08.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.562+0000 7f7babbba640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba40894d0 con 0x7f7ba40af6c0 2024-03-29T11:04:08.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.562+0000 7f7babbba640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:08.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.562+0000 7f7babbba640 10 monclient(hunting): authenticate will time out at 1004.820858s 2024-03-29T11:04:08.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.565+0000 7f7baa157640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f7ba414fdc0 0x7f7ba41ec470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:57688/0 (socket says 172.21.15.71:57688) 2024-03-29T11:04:08.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.565+0000 7f7baa157640 1 -- 172.21.15.71:0/3704336393 learned_addr learned my addr 172.21.15.71:0/3704336393 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:04:08.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.565+0000 7f7b9a7fc640 10 client.?.objecter ms_handle_connect 0x7f7ba414fdc0 2024-03-29T11:04:08.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.565+0000 7f7b9a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:08.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.566+0000 7f7b9a7fc640 1 -- 172.21.15.71:0/3704336393 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 463524503 0 0) 0x7f7ba408a630 con 0x7f7ba414fdc0 2024-03-29T11:04:08.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.566+0000 7f7b9a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:04:08.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.566+0000 7f7b9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:08.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.566+0000 7f7b9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:08.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.566+0000 7f7b9a7fc640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:04:08.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.566+0000 7f7b9a7fc640 1 -- 172.21.15.71:0/3704336393 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b88004eb0 con 0x7f7ba414fdc0 2024-03-29T11:04:08.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.566+0000 7f7b9a7fc640 1 -- 172.21.15.71:0/3704336393 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3940224439 0 0) 0x7f7b88004eb0 con 0x7f7ba414fdc0 2024-03-29T11:04:08.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.566+0000 7f7b9a7fc640 1 -- 172.21.15.71:0/3704336393 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ba408a630 con 0x7f7ba414fdc0 2024-03-29T11:04:08.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.566+0000 7f7b9a7fc640 1 -- 172.21.15.71:0/3704336393 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f7b9c001ad0 con 0x7f7ba414fdc0 2024-03-29T11:04:08.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.566+0000 7f7b9a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:08.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7b9a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:04:08.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7b9a7fc640 10 monclient(hunting): dump: 2024-03-29T11:04:08.551 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:08.551 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:08.551 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:08.551 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:08.551 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:08.552 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:08.552 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:08.552 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:08.552 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:08.552 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:08.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7b9a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:04:08.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7b9a7fc640 10 client.?.objecter ms_handle_connect 0x7f7ba40af6c0 2024-03-29T11:04:08.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7b9a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:08.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7b9a7fc640 1 -- 172.21.15.71:0/3704336393 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2716230325 0 0) 0x7f7ba408a630 con 0x7f7ba414fdc0 2024-03-29T11:04:08.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7b9a7fc640 1 -- 172.21.15.71:0/3704336393 >> v1:172.21.15.71:6790/0 conn(0x7f7ba40af6c0 legacy=0x7f7ba414e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:08.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7b9a7fc640 1 -- 172.21.15.71:0/3704336393 >> v1:172.21.15.71:6789/0 conn(0x7f7ba40ad6c0 legacy=0x7f7ba4088c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:08.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7b9a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:04:08.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7b9a7fc640 1 monclient: found mon.b 2024-03-29T11:04:08.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7b9a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:08.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7b9a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:04:08.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7b9a7fc640 1 -- 172.21.15.71:0/3704336393 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba408ade0 con 0x7f7ba414fdc0 2024-03-29T11:04:08.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7b9a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:04:08.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7b9a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:04:08.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7b9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:08.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7babbba640 5 monclient: authenticate success, global_id 4330 2024-03-29T11:04:08.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7babbba640 10 monclient: _renew_subs 2024-03-29T11:04:08.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7babbba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:04:08.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7babbba640 1 -- 172.21.15.71:0/3704336393 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba408aff0 con 0x7f7ba414fdc0 2024-03-29T11:04:08.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7babbba640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:04:08.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7babbba640 10 monclient: _renew_subs 2024-03-29T11:04:08.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7babbba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:04:08.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7babbba640 1 -- 172.21.15.71:0/3704336393 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba414ef90 con 0x7f7ba414fdc0 2024-03-29T11:04:08.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7babbba640 1 librados: init done 2024-03-29T11:04:08.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7b9a7fc640 1 -- 172.21.15.71:0/3704336393 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b9c0020a0 con 0x7f7ba414fdc0 2024-03-29T11:04:08.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7b9a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:08.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7b9a7fc640 1 -- 172.21.15.71:0/3704336393 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f7b9c004d80 con 0x7f7ba414fdc0 2024-03-29T11:04:08.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7b9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:08.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7b9a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:04:08.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.567+0000 7f7b9a7fc640 10 monclient: dump: 2024-03-29T11:04:08.554 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:08.554 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:08.554 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:08.554 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:08.554 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:08.554 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:08.554 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:08.554 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:08.554 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:08.555 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:08.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.568+0000 7f7b9b7fe640 4 set_mon_vals no callback set 2024-03-29T11:04:08.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.569+0000 7f7babbba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:08.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.569+0000 7f7b9a7fc640 1 -- 172.21.15.71:0/3704336393 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f7b9c004ee0 con 0x7f7ba414fdc0 2024-03-29T11:04:08.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.569+0000 7f7babbba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:08.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.569+0000 7f7babbba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:04:08.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.569+0000 7f7babbba640 1 -- 172.21.15.71:0/3704336393 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b8c005080 con 0x7f7ba414fdc0 2024-03-29T11:04:08.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.570+0000 7f7b9a7fc640 1 -- 172.21.15.71:0/3704336393 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f7b9c084d80 con 0x7f7ba414fdc0 2024-03-29T11:04:08.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.570+0000 7f7b9a7fc640 10 client.4330.objecter ms_dispatch 0x7f7ba4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:04:08.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.570+0000 7f7b9a7fc640 3 client.4330.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:04:08.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.570+0000 7f7b9a7fc640 3 client.4330.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:04:08.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.570+0000 7f7b9a7fc640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-29T11:04:08.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.570+0000 7f7ba9956640 10 monclient: get_auth_request con 0x7f7b8806dac0 auth_method 0 2024-03-29T11:04:08.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.571+0000 7f7b9a7fc640 10 client.4330.objecter ms_handle_connect 0x7f7b8806dac0 2024-03-29T11:04:08.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.578+0000 7f7b9a7fc640 1 -- 172.21.15.71:0/3704336393 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7b9c050660 con 0x7f7ba414fdc0 2024-03-29T11:04:08.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.578+0000 7f7b9a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:08.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.578+0000 7f7b9a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:04:08.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.914+0000 7f7babbba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:08.894 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.914+0000 7f7babbba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:08.894 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.914+0000 7f7babbba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:04:08.894 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.914+0000 7f7babbba640 1 -- 172.21.15.71:0/3704336393 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b8c005330 con 0x7f7ba414fdc0 2024-03-29T11:04:08.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.916+0000 7f7b9a7fc640 1 -- 172.21.15.71:0/3704336393 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f7b9c052590 con 0x7f7ba414fdc0 2024-03-29T11:04:08.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.916+0000 7f7b9a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:08.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.916+0000 7f7b9a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:04:08.896 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:04:08.896 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:04:08.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.920+0000 7f7babbba640 10 librados: watch_flush enter 2024-03-29T11:04:08.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.920+0000 7f7babbba640 10 librados: watch_flush exit 2024-03-29T11:04:08.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.920+0000 7f7babbba640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-29T11:04:08.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.920+0000 7f7babbba640 10 client.4330.objecter successfully canceled tick 2024-03-29T11:04:08.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.920+0000 7f7babbba640 1 -- 172.21.15.71:0/3704336393 >> v1:172.21.15.137:6816/973332665 conn(0x7f7b8806dac0 legacy=0x7f7b8806ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:08.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.920+0000 7f7babbba640 10 monclient: shutdown 2024-03-29T11:04:08.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.920+0000 7f7babbba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:08.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.920+0000 7f7babbba640 1 -- 172.21.15.71:0/3704336393 >> v1:172.21.15.137:6789/0 conn(0x7f7ba414fdc0 legacy=0x7f7ba41ec470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:08.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.920+0000 7f7babbba640 1 -- 172.21.15.71:0/3704336393 shutdown_connections 2024-03-29T11:04:08.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.920+0000 7f7babbba640 1 -- 172.21.15.71:0/3704336393 >> 172.21.15.71:0/3704336393 conn(0x7f7ba40b35e0 msgr2=0x7f7ba40a94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:08.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.922+0000 7f7babbba640 1 -- 172.21.15.71:0/3704336393 shutdown_connections 2024-03-29T11:04:08.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.923+0000 7f7babbba640 1 -- 172.21.15.71:0/3704336393 wait complete. 2024-03-29T11:04:08.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:08.924+0000 7f7babbba640 1 librados: shutdown 2024-03-29T11:04:08.919 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:04:15.725 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:04:15.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.895+0000 7f82d689e640 10 monclient: get_monmap_and_config 2024-03-29T11:04:15.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.895+0000 7f82d689e640 10 monclient: build_initial_monmap 2024-03-29T11:04:15.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.895+0000 7f82d689e640 10 monclient: monmap: 2024-03-29T11:04:15.875 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:04:15.875 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:15.875 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:04:15.895928+0000 2024-03-29T11:04:15.875 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:04:15.895928+0000 2024-03-29T11:04:15.875 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:15.875 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:15.875 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:04:15.875 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:04:15.875 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:04:15.875 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:15.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.896+0000 7f82d689e640 1 Processor -- start 2024-03-29T11:04:15.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.896+0000 7f82d689e640 1 -- start start 2024-03-29T11:04:15.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.896+0000 7f82d689e640 10 monclient: init 2024-03-29T11:04:15.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.897+0000 7f82d689e640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:15.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.897+0000 7f82d689e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:04:15.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.897+0000 7f82d689e640 10 monclient(hunting): picked mon.noname-b con 0x7f82c80389e0 addr v1:172.21.15.71:6790/0 2024-03-29T11:04:15.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.897+0000 7f82d689e640 10 monclient(hunting): picked mon.noname-a con 0x7f82c8038dc0 addr v1:172.21.15.71:6789/0 2024-03-29T11:04:15.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.897+0000 7f82d689e640 10 monclient(hunting): picked mon.noname-c con 0x7f82c8030660 addr v1:172.21.15.137:6789/0 2024-03-29T11:04:15.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.897+0000 7f82d689e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c80c54c0 con 0x7f82c8038dc0 2024-03-29T11:04:15.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.897+0000 7f82d689e640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c8036430 con 0x7f82c8030660 2024-03-29T11:04:15.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.897+0000 7f82d689e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c8039840 con 0x7f82c80389e0 2024-03-29T11:04:15.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.897+0000 7f82d689e640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:15.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.897+0000 7f82d689e640 10 monclient(hunting): authenticate will time out at 1012.155678s 2024-03-29T11:04:15.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.903+0000 7f82d509b640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f82c8038dc0 0x7f82c80cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:37613/0 (socket says 172.21.15.71:37613) 2024-03-29T11:04:15.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.903+0000 7f82d509b640 1 -- 172.21.15.71:0/1681524640 learned_addr learned my addr 172.21.15.71:0/1681524640 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:04:15.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.903+0000 7f82d489a640 1 -- 172.21.15.71:0/1681524640 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3999248372 0 0) 0x7f82c80c54c0 con 0x7f82c8038dc0 2024-03-29T11:04:15.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.903+0000 7f82d489a640 10 monclient(hunting): _negotiate 2024-03-29T11:04:15.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.903+0000 7f82d489a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:15.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.903+0000 7f82d489a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:15.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.903+0000 7f82d489a640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:04:15.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.903+0000 7f82d489a640 1 -- 172.21.15.71:0/1681524640 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82bc003520 con 0x7f82c8038dc0 2024-03-29T11:04:15.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.904+0000 7f82d489a640 1 -- 172.21.15.71:0/1681524640 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 693829831 0 0) 0x7f82c8036430 con 0x7f82c8030660 2024-03-29T11:04:15.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.904+0000 7f82d489a640 10 monclient(hunting): _negotiate 2024-03-29T11:04:15.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.904+0000 7f82d489a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:15.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.904+0000 7f82d489a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:15.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.904+0000 7f82d489a640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:04:15.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.904+0000 7f82d489a640 1 -- 172.21.15.71:0/1681524640 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82c80c54c0 con 0x7f82c8030660 2024-03-29T11:04:15.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.905+0000 7f82d489a640 1 -- 172.21.15.71:0/1681524640 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3854543497 0 0) 0x7f82bc003520 con 0x7f82c8038dc0 2024-03-29T11:04:15.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.905+0000 7f82d489a640 1 -- 172.21.15.71:0/1681524640 >> v1:172.21.15.71:6790/0 conn(0x7f82c80389e0 legacy=0x7f82c8033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:15.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.905+0000 7f82d489a640 1 -- 172.21.15.71:0/1681524640 >> v1:172.21.15.137:6789/0 conn(0x7f82c8030660 legacy=0x7f82c80d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:15.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.905+0000 7f82d489a640 10 monclient: _finish_hunting 0 2024-03-29T11:04:15.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.905+0000 7f82d489a640 1 monclient: found mon.noname-a 2024-03-29T11:04:15.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.905+0000 7f82d489a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:15.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.905+0000 7f82d489a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:04:15.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.905+0000 7f82d489a640 1 -- 172.21.15.71:0/1681524640 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c8035690 con 0x7f82c8038dc0 2024-03-29T11:04:15.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.905+0000 7f82d489a640 10 monclient: _finish_auth 0 2024-03-29T11:04:15.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.905+0000 7f82d489a640 10 monclient: _check_auth_tickets 2024-03-29T11:04:15.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.905+0000 7f82d489a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:15.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.905+0000 7f82d489a640 1 -- 172.21.15.71:0/1681524640 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f82d004f730 con 0x7f82c8038dc0 2024-03-29T11:04:15.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.905+0000 7f82d489a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:15.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.905+0000 7f82d489a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:04:15.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.905+0000 7f82d489a640 10 monclient: dump: 2024-03-29T11:04:15.886 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:15.886 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:15.886 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:15.886 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:15.886 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:15.886 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:15.886 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:15.886 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:15.887 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:15.887 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:15.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.906+0000 7f82d489a640 1 -- 172.21.15.71:0/1681524640 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82d0075f10 con 0x7f82c8038dc0 2024-03-29T11:04:15.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.906+0000 7f82d489a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:15.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.906+0000 7f82d489a640 1 -- 172.21.15.71:0/1681524640 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f82d00758b0 con 0x7f82c8038dc0 2024-03-29T11:04:15.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.906+0000 7f82d489a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:15.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.906+0000 7f82d489a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:04:15.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.906+0000 7f82d489a640 10 monclient: dump: 2024-03-29T11:04:15.887 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:15.887 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:15.887 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:15.887 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:15.888 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:15.888 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:15.888 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:15.888 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:15.888 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:15.888 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:15.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.907+0000 7f82d689e640 5 monclient: authenticate success, global_id 4377 2024-03-29T11:04:15.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.907+0000 7f82d689e640 10 monclient: get_monmap_and_config success 2024-03-29T11:04:15.889 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.907+0000 7f82d689e640 4 set_mon_vals no callback set 2024-03-29T11:04:15.889 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.907+0000 7f82d689e640 10 monclient: shutdown 2024-03-29T11:04:15.889 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.907+0000 7f82d689e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:15.889 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.907+0000 7f82d689e640 1 -- 172.21.15.71:0/1681524640 >> v1:172.21.15.71:6789/0 conn(0x7f82c8038dc0 legacy=0x7f82c80cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:15.889 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.908+0000 7f82d689e640 1 -- 172.21.15.71:0/1681524640 shutdown_connections 2024-03-29T11:04:15.889 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.908+0000 7f82d689e640 1 -- 172.21.15.71:0/1681524640 >> 172.21.15.71:0/1681524640 conn(0x7f82c8005c00 msgr2=0x7f82c8006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:15.889 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.910+0000 7f82d689e640 1 -- 172.21.15.71:0/1681524640 shutdown_connections 2024-03-29T11:04:15.890 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.910+0000 7f82d689e640 1 -- 172.21.15.71:0/1681524640 wait complete. 2024-03-29T11:04:15.890 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.911+0000 7f82d689e640 10 monclient: build_initial_monmap 2024-03-29T11:04:15.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.911+0000 7f82d689e640 10 monclient: monmap: 2024-03-29T11:04:15.891 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:04:15.891 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:04:15.891 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:04:15.891 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:04:15.891 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:15.891 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:15.891 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:04:15.891 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:04:15.891 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:04:15.891 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:15.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.911+0000 7f82d689e640 1 librados: starting msgr at 2024-03-29T11:04:15.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.911+0000 7f82d689e640 1 librados: starting objecter 2024-03-29T11:04:15.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.911+0000 7f82d689e640 1 Processor -- start 2024-03-29T11:04:15.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.911+0000 7f82d689e640 1 -- start start 2024-03-29T11:04:15.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.911+0000 7f82d689e640 1 librados: setting wanted keys 2024-03-29T11:04:15.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.911+0000 7f82d689e640 1 librados: calling monclient init 2024-03-29T11:04:15.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.911+0000 7f82d689e640 10 monclient: init 2024-03-29T11:04:15.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.912+0000 7f82d689e640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:15.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.912+0000 7f82d689e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:04:15.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.912+0000 7f82d689e640 10 monclient(hunting): picked mon.noname-b con 0x7f82c8030660 addr v1:172.21.15.137:6789/0 2024-03-29T11:04:15.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.912+0000 7f82d689e640 10 monclient(hunting): picked mon.noname-a con 0x7f82c80389e0 addr v1:172.21.15.71:6789/0 2024-03-29T11:04:15.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.916+0000 7f82d589c640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f82c8030660 0x7f82c8169420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:59333/0 (socket says 172.21.15.71:59333) 2024-03-29T11:04:15.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.919+0000 7f82d689e640 10 monclient(hunting): picked mon.noname-c con 0x7f82c816d050 addr v1:172.21.15.71:6790/0 2024-03-29T11:04:15.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.919+0000 7f82d689e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c8169c10 con 0x7f82c80389e0 2024-03-29T11:04:15.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.919+0000 7f82d689e640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c816a130 con 0x7f82c8030660 2024-03-29T11:04:15.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.919+0000 7f82d689e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c816a2c0 con 0x7f82c816d050 2024-03-29T11:04:15.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.919+0000 7f82d689e640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:15.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.919+0000 7f82d689e640 10 monclient(hunting): authenticate will time out at 1012.177514s 2024-03-29T11:04:15.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.919+0000 7f82d589c640 1 -- 172.21.15.71:0/159834458 learned_addr learned my addr 172.21.15.71:0/159834458 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:04:15.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.919+0000 7f82c67fc640 10 client.?.objecter ms_handle_connect 0x7f82c8030660 2024-03-29T11:04:15.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.919+0000 7f82c67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:15.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.919+0000 7f82c67fc640 10 client.?.objecter ms_handle_connect 0x7f82c80389e0 2024-03-29T11:04:15.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.919+0000 7f82c67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:15.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.919+0000 7f82c67fc640 10 client.?.objecter ms_handle_connect 0x7f82c816d050 2024-03-29T11:04:15.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.919+0000 7f82c67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:15.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.920+0000 7f82c67fc640 1 -- 172.21.15.71:0/159834458 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2223667425 0 0) 0x7f82c816a2c0 con 0x7f82c816d050 2024-03-29T11:04:15.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.920+0000 7f82c67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:04:15.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.920+0000 7f82c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:15.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.920+0000 7f82c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:15.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.920+0000 7f82c67fc640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:04:15.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.920+0000 7f82c67fc640 1 -- 172.21.15.71:0/159834458 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b4004eb0 con 0x7f82c816d050 2024-03-29T11:04:15.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.920+0000 7f82c67fc640 1 -- 172.21.15.71:0/159834458 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2267584677 0 0) 0x7f82b4004eb0 con 0x7f82c816d050 2024-03-29T11:04:15.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.920+0000 7f82c67fc640 1 -- 172.21.15.71:0/159834458 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82c816a2c0 con 0x7f82c816d050 2024-03-29T11:04:15.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.920+0000 7f82c67fc640 1 -- 172.21.15.71:0/159834458 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f82cc001950 con 0x7f82c816d050 2024-03-29T11:04:15.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.920+0000 7f82c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:15.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.920+0000 7f82c67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:04:15.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.920+0000 7f82c67fc640 10 monclient(hunting): dump: 2024-03-29T11:04:15.907 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:15.907 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:15.907 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:15.907 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:15.907 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:15.907 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:15.907 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:15.907 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:15.907 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:15.908 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:15.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.920+0000 7f82c67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:04:15.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.920+0000 7f82c67fc640 1 -- 172.21.15.71:0/159834458 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1043341937 0 0) 0x7f82c8169c10 con 0x7f82c80389e0 2024-03-29T11:04:15.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.920+0000 7f82c67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:04:15.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.920+0000 7f82c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:15.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.920+0000 7f82c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:15.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.920+0000 7f82c67fc640 10 monclient(hunting): my global_id is 4380 2024-03-29T11:04:15.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.920+0000 7f82c67fc640 1 -- 172.21.15.71:0/159834458 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b4004eb0 con 0x7f82c80389e0 2024-03-29T11:04:15.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.920+0000 7f82c67fc640 1 -- 172.21.15.71:0/159834458 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3257752230 0 0) 0x7f82c816a2c0 con 0x7f82c816d050 2024-03-29T11:04:15.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.920+0000 7f82c67fc640 1 -- 172.21.15.71:0/159834458 >> v1:172.21.15.137:6789/0 conn(0x7f82c8030660 legacy=0x7f82c8169420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:15.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.920+0000 7f82c67fc640 1 -- 172.21.15.71:0/159834458 >> v1:172.21.15.71:6789/0 conn(0x7f82c80389e0 legacy=0x7f82c816c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:15.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.921+0000 7f82c67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:04:15.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.921+0000 7f82c67fc640 1 monclient: found mon.c 2024-03-29T11:04:15.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.921+0000 7f82c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:15.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.921+0000 7f82c67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:04:15.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.921+0000 7f82c67fc640 1 -- 172.21.15.71:0/159834458 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c80ad030 con 0x7f82c816d050 2024-03-29T11:04:15.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.921+0000 7f82c67fc640 10 monclient: _finish_auth 0 2024-03-29T11:04:15.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.921+0000 7f82c67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:04:15.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.921+0000 7f82c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:15.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.921+0000 7f82c67fc640 1 -- 172.21.15.71:0/159834458 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82cc003610 con 0x7f82c816d050 2024-03-29T11:04:15.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.921+0000 7f82c67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:15.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.921+0000 7f82c67fc640 1 -- 172.21.15.71:0/159834458 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f82cc003a20 con 0x7f82c816d050 2024-03-29T11:04:15.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.921+0000 7f82c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:15.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.921+0000 7f82c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:04:15.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.921+0000 7f82c67fc640 10 monclient: dump: 2024-03-29T11:04:15.909 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:15.909 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:15.909 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:15.910 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:15.910 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:15.910 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:15.910 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:15.910 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:15.910 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:15.910 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:15.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.921+0000 7f82d689e640 5 monclient: authenticate success, global_id 4382 2024-03-29T11:04:15.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.921+0000 7f82d689e640 10 monclient: _renew_subs 2024-03-29T11:04:15.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.921+0000 7f82d689e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:04:15.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.921+0000 7f82d689e640 1 -- 172.21.15.71:0/159834458 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82c80ad2e0 con 0x7f82c816d050 2024-03-29T11:04:15.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.921+0000 7f82d689e640 10 client.4382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:04:15.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.921+0000 7f82d689e640 10 monclient: _renew_subs 2024-03-29T11:04:15.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.921+0000 7f82d689e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:04:15.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.921+0000 7f82d689e640 1 -- 172.21.15.71:0/159834458 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82c802f380 con 0x7f82c816d050 2024-03-29T11:04:15.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.921+0000 7f82d689e640 1 librados: init done 2024-03-29T11:04:15.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.921+0000 7f82c7fff640 4 set_mon_vals no callback set 2024-03-29T11:04:15.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.923+0000 7f82c67fc640 1 -- 172.21.15.71:0/159834458 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f82cc003b80 con 0x7f82c816d050 2024-03-29T11:04:15.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.924+0000 7f82c67fc640 1 -- 172.21.15.71:0/159834458 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f82cc083850 con 0x7f82c816d050 2024-03-29T11:04:15.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.924+0000 7f82c67fc640 10 client.4382.objecter ms_dispatch 0x7f82d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:04:15.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.924+0000 7f82c67fc640 3 client.4382.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:04:15.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.924+0000 7f82c67fc640 3 client.4382.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:04:15.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.924+0000 7f82c67fc640 20 client.4382.objecter dump_active .. 0 homeless 2024-03-29T11:04:15.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.924+0000 7f82d689e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:15.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.924+0000 7f82d689e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:15.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.924+0000 7f82d689e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:04:15.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.924+0000 7f82d689e640 1 -- 172.21.15.71:0/159834458 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b8005080 con 0x7f82c816d050 2024-03-29T11:04:15.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.924+0000 7f82d589c640 10 monclient: get_auth_request con 0x7f82b406dcf0 auth_method 0 2024-03-29T11:04:15.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.924+0000 7f82c67fc640 10 client.4382.objecter ms_handle_connect 0x7f82b406dcf0 2024-03-29T11:04:15.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.935+0000 7f82c67fc640 1 -- 172.21.15.71:0/159834458 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f82cc04f130 con 0x7f82c816d050 2024-03-29T11:04:15.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.935+0000 7f82c67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:15.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:15.935+0000 7f82c67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:04:16.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:16.263+0000 7f82d689e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:16.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:16.263+0000 7f82d689e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:16.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:16.263+0000 7f82d689e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:04:16.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:16.263+0000 7f82d689e640 1 -- 172.21.15.71:0/159834458 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82b8005330 con 0x7f82c816d050 2024-03-29T11:04:16.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:16.265+0000 7f82c67fc640 1 -- 172.21.15.71:0/159834458 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f82cc051060 con 0x7f82c816d050 2024-03-29T11:04:16.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:16.265+0000 7f82c67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:16.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:16.265+0000 7f82c67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:04:16.245 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:04:16.245 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:04:16.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:16.268+0000 7f82d689e640 10 librados: watch_flush enter 2024-03-29T11:04:16.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:16.268+0000 7f82d689e640 10 librados: watch_flush exit 2024-03-29T11:04:16.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:16.268+0000 7f82d689e640 20 client.4382.objecter shutdown clearing up homeless session... 2024-03-29T11:04:16.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:16.268+0000 7f82d689e640 10 client.4382.objecter successfully canceled tick 2024-03-29T11:04:16.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:16.268+0000 7f82d689e640 1 -- 172.21.15.71:0/159834458 >> v1:172.21.15.137:6816/973332665 conn(0x7f82b406dcf0 legacy=0x7f82b40701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:16.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:16.268+0000 7f82d689e640 10 monclient: shutdown 2024-03-29T11:04:16.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:16.268+0000 7f82d689e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:16.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:16.268+0000 7f82d689e640 1 -- 172.21.15.71:0/159834458 >> v1:172.21.15.71:6790/0 conn(0x7f82c816d050 legacy=0x7f82c81664e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:16.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:16.269+0000 7f82d689e640 1 -- 172.21.15.71:0/159834458 shutdown_connections 2024-03-29T11:04:16.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:16.269+0000 7f82d689e640 1 -- 172.21.15.71:0/159834458 >> 172.21.15.71:0/159834458 conn(0x7f82c8005c00 msgr2=0x7f82c8034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:16.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:16.269+0000 7f82d689e640 1 -- 172.21.15.71:0/159834458 shutdown_connections 2024-03-29T11:04:16.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:16.269+0000 7f82d689e640 1 -- 172.21.15.71:0/159834458 wait complete. 2024-03-29T11:04:16.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:16.269+0000 7f82d689e640 1 librados: shutdown 2024-03-29T11:04:16.265 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:04:23.073 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:04:23.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.246+0000 7f2db9f54640 10 monclient: get_monmap_and_config 2024-03-29T11:04:23.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.246+0000 7f2db9f54640 10 monclient: build_initial_monmap 2024-03-29T11:04:23.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.246+0000 7f2db9f54640 10 monclient: monmap: 2024-03-29T11:04:23.226 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:04:23.226 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:23.226 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:04:23.247048+0000 2024-03-29T11:04:23.226 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:04:23.247048+0000 2024-03-29T11:04:23.226 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:23.226 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:23.226 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:04:23.226 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:04:23.227 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:04:23.227 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:23.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.248+0000 7f2db9f54640 1 Processor -- start 2024-03-29T11:04:23.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.249+0000 7f2db9f54640 1 -- start start 2024-03-29T11:04:23.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.249+0000 7f2db9f54640 10 monclient: init 2024-03-29T11:04:23.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.249+0000 7f2db9f54640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:23.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.249+0000 7f2db9f54640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:04:23.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.249+0000 7f2db9f54640 10 monclient(hunting): picked mon.noname-c con 0x7f2db40a9ac0 addr v1:172.21.15.137:6789/0 2024-03-29T11:04:23.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.249+0000 7f2db9f54640 10 monclient(hunting): picked mon.noname-b con 0x7f2db413ee90 addr v1:172.21.15.71:6790/0 2024-03-29T11:04:23.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.249+0000 7f2db9f54640 10 monclient(hunting): picked mon.noname-a con 0x7f2db40af900 addr v1:172.21.15.71:6789/0 2024-03-29T11:04:23.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.250+0000 7f2db9f54640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db40afd10 con 0x7f2db40af900 2024-03-29T11:04:23.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.250+0000 7f2db9f54640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db40af6f0 con 0x7f2db40a9ac0 2024-03-29T11:04:23.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.250+0000 7f2db9f54640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db40aac80 con 0x7f2db413ee90 2024-03-29T11:04:23.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.250+0000 7f2db9f54640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:23.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.250+0000 7f2db9f54640 10 monclient(hunting): authenticate will time out at 1019.508138s 2024-03-29T11:04:23.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.251+0000 7f2db2ffd640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f2db413ee90 0x7f2db413b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:48318/0 (socket says 172.21.15.71:48318) 2024-03-29T11:04:23.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.251+0000 7f2db2ffd640 1 -- 172.21.15.71:0/2814741079 learned_addr learned my addr 172.21.15.71:0/2814741079 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:04:23.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.253+0000 7f2db27fc640 1 -- 172.21.15.71:0/2814741079 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1370979986 0 0) 0x7f2db40aac80 con 0x7f2db413ee90 2024-03-29T11:04:23.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.253+0000 7f2db27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:04:23.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.253+0000 7f2db27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:23.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.253+0000 7f2db27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:23.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.253+0000 7f2db27fc640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:04:23.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.253+0000 7f2db27fc640 1 -- 172.21.15.71:0/2814741079 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d9c003520 con 0x7f2db413ee90 2024-03-29T11:04:23.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.254+0000 7f2db27fc640 1 -- 172.21.15.71:0/2814741079 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837697161 0 0) 0x7f2db40af6f0 con 0x7f2db40a9ac0 2024-03-29T11:04:23.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.254+0000 7f2db27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:04:23.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.254+0000 7f2db27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:23.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.254+0000 7f2db27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:23.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.254+0000 7f2db27fc640 10 monclient(hunting): my global_id is 4339 2024-03-29T11:04:23.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.254+0000 7f2db27fc640 1 -- 172.21.15.71:0/2814741079 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2db40aac80 con 0x7f2db40a9ac0 2024-03-29T11:04:23.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.254+0000 7f2db27fc640 1 -- 172.21.15.71:0/2814741079 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 460143378 0 0) 0x7f2db40aac80 con 0x7f2db40a9ac0 2024-03-29T11:04:23.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.254+0000 7f2db27fc640 1 -- 172.21.15.71:0/2814741079 >> v1:172.21.15.71:6790/0 conn(0x7f2db413ee90 legacy=0x7f2db413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:23.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.254+0000 7f2db27fc640 1 -- 172.21.15.71:0/2814741079 >> v1:172.21.15.71:6789/0 conn(0x7f2db40af900 legacy=0x7f2db4149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:04:23.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.256+0000 7f2db27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:04:23.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.256+0000 7f2db27fc640 1 monclient: found mon.noname-c 2024-03-29T11:04:23.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.256+0000 7f2db27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:23.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.256+0000 7f2db27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.137:6789/0 2024-03-29T11:04:23.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.256+0000 7f2db27fc640 1 -- 172.21.15.71:0/2814741079 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db40aaf60 con 0x7f2db40a9ac0 2024-03-29T11:04:23.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.256+0000 7f2db27fc640 10 monclient: _finish_auth 0 2024-03-29T11:04:23.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.256+0000 7f2db27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:04:23.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.256+0000 7f2db27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:23.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.256+0000 7f2db27fc640 1 -- 172.21.15.71:0/2814741079 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2da8001670 con 0x7f2db40a9ac0 2024-03-29T11:04:23.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.256+0000 7f2db27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:23.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.256+0000 7f2db27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:04:23.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.256+0000 7f2db27fc640 10 monclient: dump: 2024-03-29T11:04:23.236 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:23.236 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:23.236 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:23.236 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:23.236 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:23.236 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:23.237 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:23.237 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:23.237 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:23.237 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:23.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.256+0000 7f2db9f54640 5 monclient: authenticate success, global_id 4339 2024-03-29T11:04:23.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.256+0000 7f2db9f54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:04:23.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.257+0000 7f2db27fc640 1 -- 172.21.15.71:0/2814741079 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2da80017d0 con 0x7f2db40a9ac0 2024-03-29T11:04:23.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.257+0000 7f2db27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:23.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.257+0000 7f2db27fc640 1 -- 172.21.15.71:0/2814741079 <== mon.1 v1:172.21.15.137:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2da8001c20 con 0x7f2db40a9ac0 2024-03-29T11:04:23.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.257+0000 7f2db27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:23.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.257+0000 7f2db27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:04:23.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.257+0000 7f2db27fc640 10 monclient: dump: 2024-03-29T11:04:23.239 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:23.239 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:23.240 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:23.240 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:23.240 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:23.240 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:23.240 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:23.240 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:23.240 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:23.240 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:23.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.257+0000 7f2db9f54640 10 monclient: get_monmap_and_config success 2024-03-29T11:04:23.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.257+0000 7f2db9f54640 4 set_mon_vals no callback set 2024-03-29T11:04:23.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.258+0000 7f2db9f54640 10 monclient: shutdown 2024-03-29T11:04:23.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.258+0000 7f2db9f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:23.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.258+0000 7f2db9f54640 1 -- 172.21.15.71:0/2814741079 >> v1:172.21.15.137:6789/0 conn(0x7f2db40a9ac0 legacy=0x7f2db40adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:23.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.260+0000 7f2db9f54640 1 -- 172.21.15.71:0/2814741079 shutdown_connections 2024-03-29T11:04:23.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.260+0000 7f2db9f54640 1 -- 172.21.15.71:0/2814741079 >> 172.21.15.71:0/2814741079 conn(0x7f2db40b35e0 msgr2=0x7f2db40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:23.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.260+0000 7f2db9f54640 1 -- 172.21.15.71:0/2814741079 shutdown_connections 2024-03-29T11:04:23.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.260+0000 7f2db9f54640 1 -- 172.21.15.71:0/2814741079 wait complete. 2024-03-29T11:04:23.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.261+0000 7f2db9f54640 10 monclient: build_initial_monmap 2024-03-29T11:04:23.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.261+0000 7f2db9f54640 10 monclient: monmap: 2024-03-29T11:04:23.244 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:04:23.244 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:04:23.244 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:04:23.244 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:04:23.244 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:23.244 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:23.244 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:04:23.244 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:04:23.245 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:04:23.245 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:23.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.262+0000 7f2db9f54640 1 librados: starting msgr at 2024-03-29T11:04:23.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.262+0000 7f2db9f54640 1 librados: starting objecter 2024-03-29T11:04:23.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.262+0000 7f2db9f54640 1 Processor -- start 2024-03-29T11:04:23.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.262+0000 7f2db9f54640 1 -- start start 2024-03-29T11:04:23.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.262+0000 7f2db9f54640 1 librados: setting wanted keys 2024-03-29T11:04:23.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.262+0000 7f2db9f54640 1 librados: calling monclient init 2024-03-29T11:04:23.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.262+0000 7f2db9f54640 10 monclient: init 2024-03-29T11:04:23.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.263+0000 7f2db9f54640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:23.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.263+0000 7f2db9f54640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:04:23.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.263+0000 7f2db9f54640 10 monclient(hunting): picked mon.noname-a con 0x7f2db40a9ac0 addr v1:172.21.15.71:6789/0 2024-03-29T11:04:23.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.263+0000 7f2db9f54640 10 monclient(hunting): picked mon.noname-c con 0x7f2db40af900 addr v1:172.21.15.71:6790/0 2024-03-29T11:04:23.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.263+0000 7f2db9f54640 10 monclient(hunting): picked mon.noname-b con 0x7f2db413ee90 addr v1:172.21.15.137:6789/0 2024-03-29T11:04:23.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.263+0000 7f2db9f54640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db40b21b0 con 0x7f2db40a9ac0 2024-03-29T11:04:23.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.263+0000 7f2db9f54640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db413de30 con 0x7f2db413ee90 2024-03-29T11:04:23.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.263+0000 7f2db9f54640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db413db00 con 0x7f2db40af900 2024-03-29T11:04:23.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.263+0000 7f2db9f54640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:23.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.263+0000 7f2db9f54640 10 monclient(hunting): authenticate will time out at 1019.522018s 2024-03-29T11:04:23.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.264+0000 7f2db2ffd640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f2db40af900 0x7f2db41d3240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:50698/0 (socket says 172.21.15.71:50698) 2024-03-29T11:04:23.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.264+0000 7f2db2ffd640 1 -- 172.21.15.71:0/2968208593 learned_addr learned my addr 172.21.15.71:0/2968208593 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:04:23.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.264+0000 7f2d9bfff640 10 client.?.objecter ms_handle_connect 0x7f2db40af900 2024-03-29T11:04:23.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.264+0000 7f2d9bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:23.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.264+0000 7f2d9bfff640 1 -- 172.21.15.71:0/2968208593 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2468824555 0 0) 0x7f2db413db00 con 0x7f2db40af900 2024-03-29T11:04:23.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.264+0000 7f2d9bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:04:23.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.264+0000 7f2d9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:23.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.264+0000 7f2d9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:23.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.264+0000 7f2d9bfff640 10 monclient(hunting): my global_id is 4388 2024-03-29T11:04:23.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.264+0000 7f2d9bfff640 1 -- 172.21.15.71:0/2968208593 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2da000d6c0 con 0x7f2db40af900 2024-03-29T11:04:23.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.267+0000 7f2d9bfff640 1 -- 172.21.15.71:0/2968208593 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 882510332 0 0) 0x7f2da000d6c0 con 0x7f2db40af900 2024-03-29T11:04:23.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.267+0000 7f2d9bfff640 1 -- 172.21.15.71:0/2968208593 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2db413db00 con 0x7f2db40af900 2024-03-29T11:04:23.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.267+0000 7f2d9bfff640 1 -- 172.21.15.71:0/2968208593 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2da4001890 con 0x7f2db40af900 2024-03-29T11:04:23.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.267+0000 7f2d9bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:23.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.267+0000 7f2d9bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:04:23.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.267+0000 7f2d9bfff640 10 monclient(hunting): dump: 2024-03-29T11:04:23.249 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:23.250 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:23.250 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:23.250 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:23.250 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:23.250 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:23.250 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:23.250 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:23.250 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:23.250 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:23.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.267+0000 7f2d9bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:04:23.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.268+0000 7f2d9bfff640 10 client.?.objecter ms_handle_connect 0x7f2db40a9ac0 2024-03-29T11:04:23.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.268+0000 7f2d9bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:23.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.268+0000 7f2d9bfff640 10 client.?.objecter ms_handle_connect 0x7f2db413ee90 2024-03-29T11:04:23.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.268+0000 7f2d9bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:23.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.269+0000 7f2d9bfff640 1 -- 172.21.15.71:0/2968208593 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814446948 0 0) 0x7f2db40b21b0 con 0x7f2db40a9ac0 2024-03-29T11:04:23.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.269+0000 7f2d9bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:04:23.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.269+0000 7f2d9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:23.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.269+0000 7f2d9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:23.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.269+0000 7f2d9bfff640 10 monclient(hunting): my global_id is 4383 2024-03-29T11:04:23.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.269+0000 7f2d9bfff640 1 -- 172.21.15.71:0/2968208593 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2da000d6c0 con 0x7f2db40a9ac0 2024-03-29T11:04:23.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.269+0000 7f2d9bfff640 1 -- 172.21.15.71:0/2968208593 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3263179574 0 0) 0x7f2db413de30 con 0x7f2db413ee90 2024-03-29T11:04:23.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.269+0000 7f2d9bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:04:23.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.269+0000 7f2d9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:23.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.269+0000 7f2d9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:23.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.269+0000 7f2d9bfff640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:04:23.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.269+0000 7f2d9bfff640 1 -- 172.21.15.71:0/2968208593 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2db40b21b0 con 0x7f2db413ee90 2024-03-29T11:04:23.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.269+0000 7f2d9bfff640 1 -- 172.21.15.71:0/2968208593 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2708281259 0 0) 0x7f2db413db00 con 0x7f2db40af900 2024-03-29T11:04:23.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.269+0000 7f2d9bfff640 1 -- 172.21.15.71:0/2968208593 >> v1:172.21.15.137:6789/0 conn(0x7f2db413ee90 legacy=0x7f2db41dd790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:23.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.270+0000 7f2d9bfff640 1 -- 172.21.15.71:0/2968208593 >> v1:172.21.15.71:6789/0 conn(0x7f2db40a9ac0 legacy=0x7f2db413d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:23.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.270+0000 7f2d9bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:04:23.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.270+0000 7f2d9bfff640 1 monclient: found mon.c 2024-03-29T11:04:23.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.270+0000 7f2d9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:23.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.270+0000 7f2d9bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:04:23.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.270+0000 7f2d9bfff640 1 -- 172.21.15.71:0/2968208593 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db413c0c0 con 0x7f2db40af900 2024-03-29T11:04:23.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.270+0000 7f2d9bfff640 10 monclient: _finish_auth 0 2024-03-29T11:04:23.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.270+0000 7f2d9bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:04:23.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.270+0000 7f2d9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:23.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.270+0000 7f2db9f54640 5 monclient: authenticate success, global_id 4388 2024-03-29T11:04:23.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.270+0000 7f2db9f54640 10 monclient: _renew_subs 2024-03-29T11:04:23.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.270+0000 7f2db9f54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:04:23.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.270+0000 7f2db9f54640 1 -- 172.21.15.71:0/2968208593 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db41d3950 con 0x7f2db40af900 2024-03-29T11:04:23.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.270+0000 7f2db9f54640 10 client.4388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:04:23.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.270+0000 7f2db9f54640 10 monclient: _renew_subs 2024-03-29T11:04:23.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.270+0000 7f2db9f54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:04:23.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.270+0000 7f2db9f54640 1 -- 172.21.15.71:0/2968208593 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db41deeb0 con 0x7f2db40af900 2024-03-29T11:04:23.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.270+0000 7f2db9f54640 1 librados: init done 2024-03-29T11:04:23.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.271+0000 7f2db9f54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:23.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.271+0000 7f2db9f54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:23.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.272+0000 7f2db9f54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:04:23.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.272+0000 7f2db9f54640 1 -- 172.21.15.71:0/2968208593 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db413bc80 con 0x7f2db40af900 2024-03-29T11:04:23.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.272+0000 7f2d9bfff640 1 -- 172.21.15.71:0/2968208593 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2da40019f0 con 0x7f2db40af900 2024-03-29T11:04:23.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.272+0000 7f2d9bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:23.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.272+0000 7f2d9bfff640 1 -- 172.21.15.71:0/2968208593 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2da4004a00 con 0x7f2db40af900 2024-03-29T11:04:23.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.272+0000 7f2d9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:23.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.272+0000 7f2d9bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:04:23.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.272+0000 7f2d9bfff640 10 monclient: dump: 2024-03-29T11:04:23.259 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:23.263 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:23.263 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:23.263 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:23.263 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:23.263 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:23.263 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:23.263 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:23.263 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:23.263 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:23.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.272+0000 7f2db1ffb640 4 set_mon_vals no callback set 2024-03-29T11:04:23.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.273+0000 7f2d9bfff640 1 -- 172.21.15.71:0/2968208593 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f2da4004b60 con 0x7f2db40af900 2024-03-29T11:04:23.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.274+0000 7f2db37fe640 10 monclient: get_auth_request con 0x7f2da0076690 auth_method 0 2024-03-29T11:04:23.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.275+0000 7f2d9bfff640 1 -- 172.21.15.71:0/2968208593 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f2da40849f0 con 0x7f2db40af900 2024-03-29T11:04:23.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.275+0000 7f2d9bfff640 10 client.4388.objecter ms_dispatch 0x7f2db4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:04:23.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.275+0000 7f2d9bfff640 3 client.4388.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:04:23.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.275+0000 7f2d9bfff640 3 client.4388.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:04:23.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.275+0000 7f2d9bfff640 20 client.4388.objecter dump_active .. 0 homeless 2024-03-29T11:04:23.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.275+0000 7f2d9bfff640 10 client.4388.objecter ms_handle_connect 0x7f2da0076690 2024-03-29T11:04:23.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.281+0000 7f2d9bfff640 1 -- 172.21.15.71:0/2968208593 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2da4050100 con 0x7f2db40af900 2024-03-29T11:04:23.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.282+0000 7f2d9bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:23.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.282+0000 7f2d9bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:04:23.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.605+0000 7f2db9f54640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:23.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.605+0000 7f2db9f54640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:23.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.605+0000 7f2db9f54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:04:23.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.605+0000 7f2db9f54640 1 -- 172.21.15.71:0/2968208593 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2db4075f10 con 0x7f2db40af900 2024-03-29T11:04:23.585 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:04:23.585 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:04:23.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.605+0000 7f2d9bfff640 1 -- 172.21.15.71:0/2968208593 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f2da4052200 con 0x7f2db40af900 2024-03-29T11:04:23.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.605+0000 7f2d9bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:23.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.605+0000 7f2d9bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:04:23.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.609+0000 7f2db9f54640 10 librados: watch_flush enter 2024-03-29T11:04:23.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.609+0000 7f2db9f54640 10 librados: watch_flush exit 2024-03-29T11:04:23.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.609+0000 7f2db9f54640 20 client.4388.objecter shutdown clearing up homeless session... 2024-03-29T11:04:23.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.609+0000 7f2db9f54640 10 client.4388.objecter successfully canceled tick 2024-03-29T11:04:23.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.609+0000 7f2db9f54640 1 -- 172.21.15.71:0/2968208593 >> v1:172.21.15.137:6816/973332665 conn(0x7f2da0076690 legacy=0x7f2da0078b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:23.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.609+0000 7f2db9f54640 10 monclient: shutdown 2024-03-29T11:04:23.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.609+0000 7f2db9f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:23.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.609+0000 7f2db9f54640 1 -- 172.21.15.71:0/2968208593 >> v1:172.21.15.71:6790/0 conn(0x7f2db40af900 legacy=0x7f2db41d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:23.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.609+0000 7f2db9f54640 1 -- 172.21.15.71:0/2968208593 shutdown_connections 2024-03-29T11:04:23.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.609+0000 7f2db9f54640 1 -- 172.21.15.71:0/2968208593 >> 172.21.15.71:0/2968208593 conn(0x7f2db40b35e0 msgr2=0x7f2db413b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:23.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.609+0000 7f2db9f54640 1 -- 172.21.15.71:0/2968208593 shutdown_connections 2024-03-29T11:04:23.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.609+0000 7f2db9f54640 1 -- 172.21.15.71:0/2968208593 wait complete. 2024-03-29T11:04:23.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:23.610+0000 7f2db9f54640 1 librados: shutdown 2024-03-29T11:04:23.607 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:04:30.414 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:04:30.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.580+0000 7f2cbed97640 10 monclient: get_monmap_and_config 2024-03-29T11:04:30.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.580+0000 7f2cbed97640 10 monclient: build_initial_monmap 2024-03-29T11:04:30.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.580+0000 7f2cbed97640 10 monclient: monmap: 2024-03-29T11:04:30.559 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:04:30.559 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:30.560 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:04:30.581150+0000 2024-03-29T11:04:30.560 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:04:30.581150+0000 2024-03-29T11:04:30.560 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:30.560 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:30.560 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:04:30.560 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:04:30.560 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:04:30.560 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:30.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.581+0000 7f2cbed97640 1 Processor -- start 2024-03-29T11:04:30.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.581+0000 7f2cbed97640 1 -- start start 2024-03-29T11:04:30.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.581+0000 7f2cbed97640 10 monclient: init 2024-03-29T11:04:30.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.582+0000 7f2cbed97640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:30.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.582+0000 7f2cbed97640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:04:30.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.582+0000 7f2cbed97640 10 monclient(hunting): picked mon.noname-c con 0x7f2cb81457f0 addr v1:172.21.15.137:6789/0 2024-03-29T11:04:30.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.582+0000 7f2cbed97640 10 monclient(hunting): picked mon.noname-a con 0x7f2cb813f650 addr v1:172.21.15.71:6789/0 2024-03-29T11:04:30.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.582+0000 7f2cbed97640 10 monclient(hunting): picked mon.noname-b con 0x7f2cb80a9d70 addr v1:172.21.15.71:6790/0 2024-03-29T11:04:30.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.582+0000 7f2cbed97640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb80aaef0 con 0x7f2cb813f650 2024-03-29T11:04:30.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.582+0000 7f2cbed97640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb8145bd0 con 0x7f2cb81457f0 2024-03-29T11:04:30.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.582+0000 7f2cbed97640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb8145eb0 con 0x7f2cb80a9d70 2024-03-29T11:04:30.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.582+0000 7f2cbed97640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:30.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.582+0000 7f2cbed97640 10 monclient(hunting): authenticate will time out at 1026.840831s 2024-03-29T11:04:30.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.584+0000 7f2cbd334640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f2cb80a9d70 0x7f2cb814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:47560/0 (socket says 172.21.15.71:47560) 2024-03-29T11:04:30.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.584+0000 7f2cbd334640 1 -- 172.21.15.71:0/2197248181 learned_addr learned my addr 172.21.15.71:0/2197248181 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:04:30.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.586+0000 7f2cb77fe640 1 -- 172.21.15.71:0/2197248181 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1620434805 0 0) 0x7f2cb8145eb0 con 0x7f2cb80a9d70 2024-03-29T11:04:30.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.586+0000 7f2cb77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:04:30.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.586+0000 7f2cb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:30.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.586+0000 7f2cb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:30.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.586+0000 7f2cb77fe640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:04:30.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.586+0000 7f2cb77fe640 1 -- 172.21.15.71:0/2197248181 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ca0003520 con 0x7f2cb80a9d70 2024-03-29T11:04:30.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.586+0000 7f2cb77fe640 1 -- 172.21.15.71:0/2197248181 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310015192 0 0) 0x7f2cb8145bd0 con 0x7f2cb81457f0 2024-03-29T11:04:30.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.586+0000 7f2cb77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:04:30.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.586+0000 7f2cb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:30.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.586+0000 7f2cb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:30.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.586+0000 7f2cb77fe640 10 monclient(hunting): my global_id is 4345 2024-03-29T11:04:30.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.586+0000 7f2cb77fe640 1 -- 172.21.15.71:0/2197248181 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cb8145eb0 con 0x7f2cb81457f0 2024-03-29T11:04:30.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.586+0000 7f2cb77fe640 1 -- 172.21.15.71:0/2197248181 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 100553256 0 0) 0x7f2cb8145eb0 con 0x7f2cb81457f0 2024-03-29T11:04:30.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.586+0000 7f2cb77fe640 1 -- 172.21.15.71:0/2197248181 >> v1:172.21.15.71:6790/0 conn(0x7f2cb80a9d70 legacy=0x7f2cb814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:30.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.586+0000 7f2cb77fe640 1 -- 172.21.15.71:0/2197248181 >> v1:172.21.15.71:6789/0 conn(0x7f2cb813f650 legacy=0x7f2cb80b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:04:30.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.588+0000 7f2cb77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:04:30.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.588+0000 7f2cb77fe640 1 monclient: found mon.noname-c 2024-03-29T11:04:30.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.588+0000 7f2cb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:30.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.588+0000 7f2cb77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.137:6789/0 2024-03-29T11:04:30.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.588+0000 7f2cb77fe640 1 -- 172.21.15.71:0/2197248181 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb8146340 con 0x7f2cb81457f0 2024-03-29T11:04:30.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.588+0000 7f2cb77fe640 10 monclient: _finish_auth 0 2024-03-29T11:04:30.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.588+0000 7f2cb77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:04:30.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.588+0000 7f2cb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:30.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.588+0000 7f2cb77fe640 1 -- 172.21.15.71:0/2197248181 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2ca80015a0 con 0x7f2cb81457f0 2024-03-29T11:04:30.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.588+0000 7f2cb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:30.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.588+0000 7f2cb77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:04:30.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.588+0000 7f2cb77fe640 10 monclient: dump: 2024-03-29T11:04:30.569 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:30.569 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:30.569 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:30.569 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:30.569 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:30.570 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:30.570 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:30.570 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:30.570 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:30.570 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:30.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.588+0000 7f2cbed97640 5 monclient: authenticate success, global_id 4345 2024-03-29T11:04:30.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.588+0000 7f2cbed97640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:04:30.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.589+0000 7f2cb77fe640 1 -- 172.21.15.71:0/2197248181 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ca8001700 con 0x7f2cb81457f0 2024-03-29T11:04:30.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.589+0000 7f2cb77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:30.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.589+0000 7f2cbed97640 10 monclient: get_monmap_and_config success 2024-03-29T11:04:30.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.589+0000 7f2cbed97640 4 set_mon_vals no callback set 2024-03-29T11:04:30.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.589+0000 7f2cb77fe640 1 -- 172.21.15.71:0/2197248181 <== mon.1 v1:172.21.15.137:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2ca8001b50 con 0x7f2cb81457f0 2024-03-29T11:04:30.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.589+0000 7f2cb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:30.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.589+0000 7f2cb77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:04:30.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.589+0000 7f2cb77fe640 10 monclient: dump: 2024-03-29T11:04:30.571 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:30.571 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:30.571 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:30.571 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:30.571 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:30.571 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:30.571 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:30.571 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:30.571 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:30.571 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:30.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.591+0000 7f2cbed97640 10 monclient: shutdown 2024-03-29T11:04:30.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.591+0000 7f2cbed97640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:30.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.591+0000 7f2cbed97640 1 -- 172.21.15.71:0/2197248181 >> v1:172.21.15.137:6789/0 conn(0x7f2cb81457f0 legacy=0x7f2cb813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:30.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.593+0000 7f2cbed97640 1 -- 172.21.15.71:0/2197248181 shutdown_connections 2024-03-29T11:04:30.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.593+0000 7f2cbed97640 1 -- 172.21.15.71:0/2197248181 >> 172.21.15.71:0/2197248181 conn(0x7f2cb8066a00 msgr2=0x7f2cb8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:30.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.593+0000 7f2cbed97640 1 -- 172.21.15.71:0/2197248181 shutdown_connections 2024-03-29T11:04:30.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.594+0000 7f2cbed97640 1 -- 172.21.15.71:0/2197248181 wait complete. 2024-03-29T11:04:30.574 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.594+0000 7f2cbed97640 10 monclient: build_initial_monmap 2024-03-29T11:04:30.574 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.594+0000 7f2cbed97640 10 monclient: monmap: 2024-03-29T11:04:30.575 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:04:30.575 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:04:30.575 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:04:30.575 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:04:30.575 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:30.575 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:30.575 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:04:30.575 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:04:30.575 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:04:30.575 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:30.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.595+0000 7f2cbed97640 1 librados: starting msgr at 2024-03-29T11:04:30.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.595+0000 7f2cbed97640 1 librados: starting objecter 2024-03-29T11:04:30.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.595+0000 7f2cbed97640 1 Processor -- start 2024-03-29T11:04:30.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.595+0000 7f2cbed97640 1 -- start start 2024-03-29T11:04:30.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.595+0000 7f2cbed97640 1 librados: setting wanted keys 2024-03-29T11:04:30.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.595+0000 7f2cbed97640 1 librados: calling monclient init 2024-03-29T11:04:30.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.595+0000 7f2cbed97640 10 monclient: init 2024-03-29T11:04:30.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.596+0000 7f2cbed97640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:30.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.596+0000 7f2cbed97640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:04:30.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.596+0000 7f2cbed97640 10 monclient(hunting): picked mon.noname-b con 0x7f2cb80a9d70 addr v1:172.21.15.137:6789/0 2024-03-29T11:04:30.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.596+0000 7f2cbed97640 10 monclient(hunting): picked mon.noname-c con 0x7f2cb813f650 addr v1:172.21.15.71:6790/0 2024-03-29T11:04:30.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.596+0000 7f2cbed97640 10 monclient(hunting): picked mon.noname-a con 0x7f2cb81457f0 addr v1:172.21.15.71:6789/0 2024-03-29T11:04:30.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.596+0000 7f2cbed97640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb8142e40 con 0x7f2cb81457f0 2024-03-29T11:04:30.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.596+0000 7f2cbed97640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb806fa70 con 0x7f2cb80a9d70 2024-03-29T11:04:30.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.596+0000 7f2cbed97640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb8138780 con 0x7f2cb813f650 2024-03-29T11:04:30.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.596+0000 7f2cbed97640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:30.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.596+0000 7f2cbed97640 10 monclient(hunting): authenticate will time out at 1026.854775s 2024-03-29T11:04:30.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.598+0000 7f2cb7fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f2cb813f650 0x7f2cb813ff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:51048/0 (socket says 172.21.15.71:51048) 2024-03-29T11:04:30.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.598+0000 7f2cb7fff640 1 -- 172.21.15.71:0/3092042215 learned_addr learned my addr 172.21.15.71:0/3092042215 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:04:30.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.598+0000 7f2cb57fa640 10 client.?.objecter ms_handle_connect 0x7f2cb813f650 2024-03-29T11:04:30.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.598+0000 7f2cb57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:30.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.598+0000 7f2cb57fa640 1 -- 172.21.15.71:0/3092042215 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2093977955 0 0) 0x7f2cb8138780 con 0x7f2cb813f650 2024-03-29T11:04:30.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.598+0000 7f2cb57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:04:30.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.598+0000 7f2cb57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:30.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.599+0000 7f2cb57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:30.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.599+0000 7f2cb57fa640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:04:30.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.599+0000 7f2cb57fa640 1 -- 172.21.15.71:0/3092042215 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ca4004eb0 con 0x7f2cb813f650 2024-03-29T11:04:30.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.600+0000 7f2cb57fa640 10 client.?.objecter ms_handle_connect 0x7f2cb81457f0 2024-03-29T11:04:30.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.600+0000 7f2cb57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:30.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.601+0000 7f2cb57fa640 10 client.?.objecter ms_handle_connect 0x7f2cb80a9d70 2024-03-29T11:04:30.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.601+0000 7f2cb57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:30.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.601+0000 7f2cb57fa640 1 -- 172.21.15.71:0/3092042215 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4244845498 0 0) 0x7f2ca4004eb0 con 0x7f2cb813f650 2024-03-29T11:04:30.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.601+0000 7f2cb57fa640 1 -- 172.21.15.71:0/3092042215 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cb8138780 con 0x7f2cb813f650 2024-03-29T11:04:30.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.601+0000 7f2cb57fa640 1 -- 172.21.15.71:0/3092042215 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2cac001960 con 0x7f2cb813f650 2024-03-29T11:04:30.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.601+0000 7f2cb57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:30.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.601+0000 7f2cb57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:04:30.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.601+0000 7f2cb57fa640 10 monclient(hunting): dump: 2024-03-29T11:04:30.586 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:30.586 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:30.587 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:30.587 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:30.587 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:30.587 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:30.587 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:30.587 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:30.587 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:30.587 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:30.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.601+0000 7f2cb57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:04:30.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.601+0000 7f2cb57fa640 1 -- 172.21.15.71:0/3092042215 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4014862297 0 0) 0x7f2ca0003520 con 0x7f2cb81457f0 2024-03-29T11:04:30.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.601+0000 7f2cb57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:04:30.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.601+0000 7f2cb57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:30.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.601+0000 7f2cb57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:30.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.601+0000 7f2cb57fa640 10 monclient(hunting): my global_id is 4386 2024-03-29T11:04:30.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.601+0000 7f2cb57fa640 1 -- 172.21.15.71:0/3092042215 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ca4004eb0 con 0x7f2cb81457f0 2024-03-29T11:04:30.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.601+0000 7f2cb57fa640 1 -- 172.21.15.71:0/3092042215 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2160554255 0 0) 0x7f2cb806fa70 con 0x7f2cb80a9d70 2024-03-29T11:04:30.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.602+0000 7f2cb57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:04:30.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.602+0000 7f2cb57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:30.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.602+0000 7f2cb57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:30.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.602+0000 7f2cb57fa640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:04:30.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.602+0000 7f2cb57fa640 1 -- 172.21.15.71:0/3092042215 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ca0003520 con 0x7f2cb80a9d70 2024-03-29T11:04:30.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.602+0000 7f2cb57fa640 1 -- 172.21.15.71:0/3092042215 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3903707209 0 0) 0x7f2ca4004eb0 con 0x7f2cb81457f0 2024-03-29T11:04:30.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.602+0000 7f2cb57fa640 1 -- 172.21.15.71:0/3092042215 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cb806fa70 con 0x7f2cb81457f0 2024-03-29T11:04:30.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.602+0000 7f2cb57fa640 1 -- 172.21.15.71:0/3092042215 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2cb0001ab0 con 0x7f2cb81457f0 2024-03-29T11:04:30.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.602+0000 7f2cb57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:30.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.602+0000 7f2cb57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:04:30.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.602+0000 7f2cb57fa640 10 monclient(hunting): dump: 2024-03-29T11:04:30.588 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:30.588 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:30.589 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:30.589 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:30.589 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:30.589 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:30.589 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:30.589 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:30.589 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:30.589 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:30.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.602+0000 7f2cb57fa640 1 -- 172.21.15.71:0/3092042215 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3236835755 0 0) 0x7f2cb806fa70 con 0x7f2cb81457f0 2024-03-29T11:04:30.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.602+0000 7f2cb57fa640 1 -- 172.21.15.71:0/3092042215 >> v1:172.21.15.71:6790/0 conn(0x7f2cb813f650 legacy=0x7f2cb813ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:30.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.602+0000 7f2cb57fa640 1 -- 172.21.15.71:0/3092042215 >> v1:172.21.15.137:6789/0 conn(0x7f2cb80a9d70 legacy=0x7f2cb806e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:30.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.602+0000 7f2cb57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:04:30.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.602+0000 7f2cb57fa640 1 monclient: found mon.a 2024-03-29T11:04:30.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.603+0000 7f2cb57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:30.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.603+0000 7f2cb57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:04:30.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.603+0000 7f2cb57fa640 1 -- 172.21.15.71:0/3092042215 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb8138910 con 0x7f2cb81457f0 2024-03-29T11:04:30.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.603+0000 7f2cb57fa640 10 monclient: _finish_auth 0 2024-03-29T11:04:30.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.603+0000 7f2cb57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:04:30.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.603+0000 7f2cb57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:30.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.603+0000 7f2cbed97640 5 monclient: authenticate success, global_id 4386 2024-03-29T11:04:30.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.603+0000 7f2cbed97640 10 monclient: _renew_subs 2024-03-29T11:04:30.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.603+0000 7f2cbed97640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:04:30.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.603+0000 7f2cbed97640 1 -- 172.21.15.71:0/3092042215 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb806e810 con 0x7f2cb81457f0 2024-03-29T11:04:30.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.603+0000 7f2cbed97640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:04:30.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.603+0000 7f2cbed97640 10 monclient: _renew_subs 2024-03-29T11:04:30.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.603+0000 7f2cbed97640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:04:30.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.603+0000 7f2cbed97640 1 -- 172.21.15.71:0/3092042215 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb8140800 con 0x7f2cb81457f0 2024-03-29T11:04:30.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.603+0000 7f2cbed97640 1 librados: init done 2024-03-29T11:04:30.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.604+0000 7f2cbed97640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:30.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.604+0000 7f2cbed97640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:30.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.604+0000 7f2cbed97640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:04:30.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.604+0000 7f2cbed97640 1 -- 172.21.15.71:0/3092042215 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cb8075f10 con 0x7f2cb81457f0 2024-03-29T11:04:30.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.604+0000 7f2cb57fa640 1 -- 172.21.15.71:0/3092042215 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2cb0001f80 con 0x7f2cb81457f0 2024-03-29T11:04:30.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.604+0000 7f2cb57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:30.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.604+0000 7f2cb6ffd640 4 set_mon_vals no callback set 2024-03-29T11:04:30.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.606+0000 7f2cb57fa640 1 -- 172.21.15.71:0/3092042215 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2cb0004d00 con 0x7f2cb81457f0 2024-03-29T11:04:30.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.606+0000 7f2cb57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:30.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.606+0000 7f2cb57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:04:30.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.607+0000 7f2cb57fa640 10 monclient: dump: 2024-03-29T11:04:30.591 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:30.591 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:30.591 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:30.591 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:30.591 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:30.591 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:30.592 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:30.592 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:30.592 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:30.592 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:30.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.607+0000 7f2cb57fa640 1 -- 172.21.15.71:0/3092042215 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f2cb001acd0 con 0x7f2cb81457f0 2024-03-29T11:04:30.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.607+0000 7f2cb57fa640 1 -- 172.21.15.71:0/3092042215 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f2cb00849f0 con 0x7f2cb81457f0 2024-03-29T11:04:30.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.607+0000 7f2cb57fa640 10 client.4386.objecter ms_dispatch 0x7f2cb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:04:30.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.607+0000 7f2cb57fa640 3 client.4386.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:04:30.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.607+0000 7f2cb57fa640 3 client.4386.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:04:30.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.608+0000 7f2cb57fa640 20 client.4386.objecter dump_active .. 0 homeless 2024-03-29T11:04:30.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.608+0000 7f2cbcb33640 10 monclient: get_auth_request con 0x7f2ca406dd60 auth_method 0 2024-03-29T11:04:30.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.609+0000 7f2cb57fa640 10 client.4386.objecter ms_handle_connect 0x7f2ca406dd60 2024-03-29T11:04:30.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.613+0000 7f2cb57fa640 1 -- 172.21.15.71:0/3092042215 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2cb0050350 con 0x7f2cb81457f0 2024-03-29T11:04:30.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.613+0000 7f2cb57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:30.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.613+0000 7f2cb57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:04:30.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.931+0000 7f2cbed97640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:30.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.931+0000 7f2cbed97640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:30.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.931+0000 7f2cbed97640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:04:30.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.931+0000 7f2cbed97640 1 -- 172.21.15.71:0/3092042215 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2cb80b3680 con 0x7f2cb81457f0 2024-03-29T11:04:30.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.933+0000 7f2cb57fa640 1 -- 172.21.15.71:0/3092042215 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f2cb0052280 con 0x7f2cb81457f0 2024-03-29T11:04:30.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.933+0000 7f2cb57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:30.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.933+0000 7f2cb57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:04:30.916 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:04:30.916 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:04:30.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.937+0000 7f2cbed97640 10 librados: watch_flush enter 2024-03-29T11:04:30.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.937+0000 7f2cbed97640 10 librados: watch_flush exit 2024-03-29T11:04:30.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.937+0000 7f2cbed97640 20 client.4386.objecter shutdown clearing up homeless session... 2024-03-29T11:04:30.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.937+0000 7f2cbed97640 10 client.4386.objecter successfully canceled tick 2024-03-29T11:04:30.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.937+0000 7f2cbed97640 1 -- 172.21.15.71:0/3092042215 >> v1:172.21.15.137:6816/973332665 conn(0x7f2ca406dd60 legacy=0x7f2ca4070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:30.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.937+0000 7f2cbed97640 10 monclient: shutdown 2024-03-29T11:04:30.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.937+0000 7f2cbed97640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:30.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.937+0000 7f2cbed97640 1 -- 172.21.15.71:0/3092042215 >> v1:172.21.15.71:6789/0 conn(0x7f2cb81457f0 legacy=0x7f2cb81e5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:30.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.938+0000 7f2cbed97640 1 -- 172.21.15.71:0/3092042215 shutdown_connections 2024-03-29T11:04:30.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.938+0000 7f2cbed97640 1 -- 172.21.15.71:0/3092042215 >> 172.21.15.71:0/3092042215 conn(0x7f2cb8066a00 msgr2=0x7f2cb813c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:30.918 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.939+0000 7f2cbed97640 1 -- 172.21.15.71:0/3092042215 shutdown_connections 2024-03-29T11:04:30.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.939+0000 7f2cbed97640 1 -- 172.21.15.71:0/3092042215 wait complete. 2024-03-29T11:04:30.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:30.940+0000 7f2cbed97640 1 librados: shutdown 2024-03-29T11:04:30.935 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:04:37.743 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:04:37.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.917+0000 7ff3f1814640 10 monclient: get_monmap_and_config 2024-03-29T11:04:37.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.917+0000 7ff3f1814640 10 monclient: build_initial_monmap 2024-03-29T11:04:37.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.917+0000 7ff3f1814640 10 monclient: monmap: 2024-03-29T11:04:37.898 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:04:37.899 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:37.899 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:04:37.919277+0000 2024-03-29T11:04:37.899 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:04:37.919277+0000 2024-03-29T11:04:37.899 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:37.899 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:37.899 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:04:37.899 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:04:37.899 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:04:37.899 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:37.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.918+0000 7ff3f1814640 1 Processor -- start 2024-03-29T11:04:37.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.918+0000 7ff3f1814640 1 -- start start 2024-03-29T11:04:37.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.918+0000 7ff3f1814640 10 monclient: init 2024-03-29T11:04:37.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.919+0000 7ff3f1814640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:37.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.919+0000 7ff3f1814640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:04:37.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.919+0000 7ff3f1814640 10 monclient(hunting): picked mon.noname-b con 0x7ff3ec0b0770 addr v1:172.21.15.71:6790/0 2024-03-29T11:04:37.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.922+0000 7ff3f1814640 10 monclient(hunting): picked mon.noname-c con 0x7ff3ec14a700 addr v1:172.21.15.137:6789/0 2024-03-29T11:04:37.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.922+0000 7ff3f1814640 10 monclient(hunting): picked mon.noname-a con 0x7ff3ec0ac5a0 addr v1:172.21.15.71:6789/0 2024-03-29T11:04:37.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.922+0000 7ff3f1814640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ec0aba70 con 0x7ff3ec0ac5a0 2024-03-29T11:04:37.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.922+0000 7ff3f1814640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ec0aecb0 con 0x7ff3ec14a700 2024-03-29T11:04:37.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.922+0000 7ff3eaffd640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7ff3ec0b0770 0x7ff3ec149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:39625/0 (socket says 172.21.15.71:39625) 2024-03-29T11:04:37.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.922+0000 7ff3eaffd640 1 -- 172.21.15.71:0/3993516828 learned_addr learned my addr 172.21.15.71:0/3993516828 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:04:37.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.923+0000 7ff3f1814640 1 -- 172.21.15.71:0/3993516828 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ec0af150 con 0x7ff3ec0b0770 2024-03-29T11:04:37.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.923+0000 7ff3f1814640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:37.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.923+0000 7ff3f1814640 10 monclient(hunting): authenticate will time out at 1034.182384s 2024-03-29T11:04:37.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.925+0000 7ff3e9ffb640 1 -- 172.21.15.71:0/3993516828 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3406407192 0 0) 0x7ff3ec0af150 con 0x7ff3ec0b0770 2024-03-29T11:04:37.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.925+0000 7ff3e9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:04:37.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.925+0000 7ff3e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:37.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.925+0000 7ff3e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:37.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.925+0000 7ff3e9ffb640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:04:37.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.925+0000 7ff3e9ffb640 1 -- 172.21.15.71:0/3993516828 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d4003520 con 0x7ff3ec0b0770 2024-03-29T11:04:37.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.926+0000 7ff3e9ffb640 1 -- 172.21.15.71:0/3993516828 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3337599998 0 0) 0x7ff3ec0aecb0 con 0x7ff3ec14a700 2024-03-29T11:04:37.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.926+0000 7ff3e9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:04:37.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.926+0000 7ff3e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:37.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.926+0000 7ff3e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:37.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.926+0000 7ff3e9ffb640 10 monclient(hunting): my global_id is 4351 2024-03-29T11:04:37.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.926+0000 7ff3e9ffb640 1 -- 172.21.15.71:0/3993516828 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3ec0af150 con 0x7ff3ec14a700 2024-03-29T11:04:37.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.926+0000 7ff3e9ffb640 1 -- 172.21.15.71:0/3993516828 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1760614368 0 0) 0x7ff3ec0af150 con 0x7ff3ec14a700 2024-03-29T11:04:37.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.926+0000 7ff3e9ffb640 1 -- 172.21.15.71:0/3993516828 >> v1:172.21.15.71:6790/0 conn(0x7ff3ec0b0770 legacy=0x7ff3ec149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:37.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.926+0000 7ff3e9ffb640 1 -- 172.21.15.71:0/3993516828 >> v1:172.21.15.71:6789/0 conn(0x7ff3ec0ac5a0 legacy=0x7ff3ec151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:04:37.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.927+0000 7ff3e9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:04:37.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.927+0000 7ff3e9ffb640 1 monclient: found mon.noname-c 2024-03-29T11:04:37.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.927+0000 7ff3e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:37.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.927+0000 7ff3e9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.137:6789/0 2024-03-29T11:04:37.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.927+0000 7ff3e9ffb640 1 -- 172.21.15.71:0/3993516828 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ec0ac9d0 con 0x7ff3ec14a700 2024-03-29T11:04:37.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.927+0000 7ff3e9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:04:37.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.927+0000 7ff3e9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:04:37.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.927+0000 7ff3e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:37.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.927+0000 7ff3e9ffb640 1 -- 172.21.15.71:0/3993516828 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7ff3dc0015a0 con 0x7ff3ec14a700 2024-03-29T11:04:37.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.927+0000 7ff3e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:37.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.928+0000 7ff3e9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:04:37.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.928+0000 7ff3e9ffb640 10 monclient: dump: 2024-03-29T11:04:37.911 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:37.911 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:37.911 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:37.911 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:37.911 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:37.911 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:37.911 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:37.912 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:37.912 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:37.912 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:37.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.929+0000 7ff3f1814640 5 monclient: authenticate success, global_id 4351 2024-03-29T11:04:37.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.929+0000 7ff3f1814640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:04:37.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.930+0000 7ff3e9ffb640 1 -- 172.21.15.71:0/3993516828 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3dc001780 con 0x7ff3ec14a700 2024-03-29T11:04:37.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.930+0000 7ff3e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:37.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.930+0000 7ff3e9ffb640 1 -- 172.21.15.71:0/3993516828 <== mon.1 v1:172.21.15.137:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7ff3dc001bd0 con 0x7ff3ec14a700 2024-03-29T11:04:37.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.930+0000 7ff3e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:37.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.930+0000 7ff3e9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:04:37.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.930+0000 7ff3e9ffb640 10 monclient: dump: 2024-03-29T11:04:37.914 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:37.914 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:37.914 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:37.914 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:37.914 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:37.914 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:37.914 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:37.914 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:37.915 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:37.915 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:37.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.930+0000 7ff3f1814640 10 monclient: get_monmap_and_config success 2024-03-29T11:04:37.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.930+0000 7ff3f1814640 4 set_mon_vals no callback set 2024-03-29T11:04:37.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.931+0000 7ff3f1814640 10 monclient: shutdown 2024-03-29T11:04:37.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.931+0000 7ff3f1814640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:37.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.931+0000 7ff3f1814640 1 -- 172.21.15.71:0/3993516828 >> v1:172.21.15.137:6789/0 conn(0x7ff3ec14a700 legacy=0x7ff3ec0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:37.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.931+0000 7ff3f1814640 1 -- 172.21.15.71:0/3993516828 shutdown_connections 2024-03-29T11:04:37.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.931+0000 7ff3f1814640 1 -- 172.21.15.71:0/3993516828 >> 172.21.15.71:0/3993516828 conn(0x7ff3ec06b660 msgr2=0x7ff3ec06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:37.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.933+0000 7ff3f1814640 1 -- 172.21.15.71:0/3993516828 shutdown_connections 2024-03-29T11:04:37.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.933+0000 7ff3f1814640 1 -- 172.21.15.71:0/3993516828 wait complete. 2024-03-29T11:04:37.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.934+0000 7ff3f1814640 10 monclient: build_initial_monmap 2024-03-29T11:04:37.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.934+0000 7ff3f1814640 10 monclient: monmap: 2024-03-29T11:04:37.917 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:04:37.917 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:04:37.917 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:04:37.917 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:04:37.917 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:37.917 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:37.917 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:04:37.917 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:04:37.917 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:04:37.917 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:37.918 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.935+0000 7ff3f1814640 1 librados: starting msgr at 2024-03-29T11:04:37.918 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.935+0000 7ff3f1814640 1 librados: starting objecter 2024-03-29T11:04:37.918 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.935+0000 7ff3f1814640 1 Processor -- start 2024-03-29T11:04:37.918 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.935+0000 7ff3f1814640 1 -- start start 2024-03-29T11:04:37.918 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.935+0000 7ff3f1814640 1 librados: setting wanted keys 2024-03-29T11:04:37.918 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.935+0000 7ff3f1814640 1 librados: calling monclient init 2024-03-29T11:04:37.918 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.935+0000 7ff3f1814640 10 monclient: init 2024-03-29T11:04:37.918 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.936+0000 7ff3f1814640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:37.918 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.936+0000 7ff3f1814640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:04:37.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.939+0000 7ff3f1814640 10 monclient(hunting): picked mon.noname-c con 0x7ff3ec0ac5a0 addr v1:172.21.15.71:6790/0 2024-03-29T11:04:37.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.939+0000 7ff3f1814640 10 monclient(hunting): picked mon.noname-b con 0x7ff3ec0b0770 addr v1:172.21.15.137:6789/0 2024-03-29T11:04:37.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.939+0000 7ff3f1814640 10 monclient(hunting): picked mon.noname-a con 0x7ff3ec14a700 addr v1:172.21.15.71:6789/0 2024-03-29T11:04:37.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.939+0000 7ff3f1814640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ec126220 con 0x7ff3ec14a700 2024-03-29T11:04:37.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.939+0000 7ff3f1814640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ec145ac0 con 0x7ff3ec0b0770 2024-03-29T11:04:37.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.939+0000 7ff3f1814640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ec145c50 con 0x7ff3ec0ac5a0 2024-03-29T11:04:37.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.939+0000 7ff3f1814640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:37.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.939+0000 7ff3f1814640 10 monclient(hunting): authenticate will time out at 1034.198400s 2024-03-29T11:04:37.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.939+0000 7ff3eaffd640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7ff3ec0ac5a0 0x7ff3ec1266b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:39737/0 (socket says 172.21.15.71:39737) 2024-03-29T11:04:37.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.939+0000 7ff3eaffd640 1 -- 172.21.15.71:0/478867030 learned_addr learned my addr 172.21.15.71:0/478867030 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:04:37.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.943+0000 7ff3d3fff640 10 client.?.objecter ms_handle_connect 0x7ff3ec0ac5a0 2024-03-29T11:04:37.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.943+0000 7ff3d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:37.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.944+0000 7ff3d3fff640 10 client.?.objecter ms_handle_connect 0x7ff3ec0b0770 2024-03-29T11:04:37.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.944+0000 7ff3d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:37.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.945+0000 7ff3d3fff640 1 -- 172.21.15.71:0/478867030 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261563280 0 0) 0x7ff3ec145ac0 con 0x7ff3ec0b0770 2024-03-29T11:04:37.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.945+0000 7ff3d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:04:37.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.945+0000 7ff3d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:37.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.945+0000 7ff3d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:37.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.945+0000 7ff3d3fff640 10 monclient(hunting): my global_id is 4354 2024-03-29T11:04:37.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.945+0000 7ff3d3fff640 1 -- 172.21.15.71:0/478867030 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d8004eb0 con 0x7ff3ec0b0770 2024-03-29T11:04:37.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.945+0000 7ff3d3fff640 10 client.?.objecter ms_handle_connect 0x7ff3ec14a700 2024-03-29T11:04:37.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.945+0000 7ff3d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:37.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.945+0000 7ff3d3fff640 1 -- 172.21.15.71:0/478867030 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171020038 0 0) 0x7ff3ec126220 con 0x7ff3ec14a700 2024-03-29T11:04:37.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.945+0000 7ff3d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:04:37.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.945+0000 7ff3d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:37.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.945+0000 7ff3d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:37.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.945+0000 7ff3d3fff640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:04:37.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.945+0000 7ff3d3fff640 1 -- 172.21.15.71:0/478867030 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3ec145ac0 con 0x7ff3ec14a700 2024-03-29T11:04:37.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.945+0000 7ff3d3fff640 1 -- 172.21.15.71:0/478867030 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1926593226 0 0) 0x7ff3d8004eb0 con 0x7ff3ec0b0770 2024-03-29T11:04:37.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.945+0000 7ff3d3fff640 1 -- 172.21.15.71:0/478867030 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3ec126220 con 0x7ff3ec0b0770 2024-03-29T11:04:37.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.945+0000 7ff3d3fff640 1 -- 172.21.15.71:0/478867030 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7ff3dc001bd0 con 0x7ff3ec0b0770 2024-03-29T11:04:37.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.945+0000 7ff3d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:37.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.945+0000 7ff3d3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:04:37.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.945+0000 7ff3d3fff640 10 monclient(hunting): dump: 2024-03-29T11:04:37.932 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:37.932 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:37.932 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:37.932 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:37.932 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:37.932 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:37.932 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:37.932 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:37.932 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:37.932 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:37.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.945+0000 7ff3d3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:04:37.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.945+0000 7ff3d3fff640 1 -- 172.21.15.71:0/478867030 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1870894314 0 0) 0x7ff3ec145ac0 con 0x7ff3ec14a700 2024-03-29T11:04:37.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.946+0000 7ff3d3fff640 1 -- 172.21.15.71:0/478867030 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3d8004eb0 con 0x7ff3ec14a700 2024-03-29T11:04:37.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.946+0000 7ff3d3fff640 1 -- 172.21.15.71:0/478867030 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7ff3e0001760 con 0x7ff3ec14a700 2024-03-29T11:04:37.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.946+0000 7ff3d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:37.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.946+0000 7ff3d3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:04:37.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.946+0000 7ff3d3fff640 10 monclient(hunting): dump: 2024-03-29T11:04:37.933 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:37.933 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:37.933 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:37.933 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:37.933 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:37.933 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:37.933 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:37.933 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:37.933 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:37.933 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:37.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.946+0000 7ff3d3fff640 1 -- 172.21.15.71:0/478867030 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1468651759 0 0) 0x7ff3ec145c50 con 0x7ff3ec0ac5a0 2024-03-29T11:04:37.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.946+0000 7ff3d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:04:37.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.946+0000 7ff3d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:37.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.946+0000 7ff3d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:37.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.946+0000 7ff3d3fff640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:04:37.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.946+0000 7ff3d3fff640 1 -- 172.21.15.71:0/478867030 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3ec145ac0 con 0x7ff3ec0ac5a0 2024-03-29T11:04:37.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.946+0000 7ff3d3fff640 1 -- 172.21.15.71:0/478867030 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 666700225 0 0) 0x7ff3ec126220 con 0x7ff3ec0b0770 2024-03-29T11:04:37.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.946+0000 7ff3d3fff640 1 -- 172.21.15.71:0/478867030 >> v1:172.21.15.71:6790/0 conn(0x7ff3ec0ac5a0 legacy=0x7ff3ec1266b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:37.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.946+0000 7ff3d3fff640 1 -- 172.21.15.71:0/478867030 >> v1:172.21.15.71:6789/0 conn(0x7ff3ec14a700 legacy=0x7ff3ec1e5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:37.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.946+0000 7ff3d3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:04:37.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.946+0000 7ff3d3fff640 1 monclient: found mon.b 2024-03-29T11:04:37.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.946+0000 7ff3d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:37.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.946+0000 7ff3d3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:04:37.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.946+0000 7ff3d3fff640 1 -- 172.21.15.71:0/478867030 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ec125cb0 con 0x7ff3ec0b0770 2024-03-29T11:04:37.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.946+0000 7ff3d3fff640 10 monclient: _finish_auth 0 2024-03-29T11:04:37.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.946+0000 7ff3d3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:04:37.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.946+0000 7ff3d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:37.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.947+0000 7ff3f1814640 5 monclient: authenticate success, global_id 4354 2024-03-29T11:04:37.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.947+0000 7ff3f1814640 10 monclient: _renew_subs 2024-03-29T11:04:37.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.947+0000 7ff3f1814640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:04:37.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.947+0000 7ff3f1814640 1 -- 172.21.15.71:0/478867030 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3ec125f60 con 0x7ff3ec0b0770 2024-03-29T11:04:37.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.947+0000 7ff3f1814640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:04:37.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.947+0000 7ff3f1814640 10 monclient: _renew_subs 2024-03-29T11:04:37.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.947+0000 7ff3f1814640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:04:37.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.947+0000 7ff3f1814640 1 -- 172.21.15.71:0/478867030 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ec1db600 con 0x7ff3ec0b0770 2024-03-29T11:04:37.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.947+0000 7ff3f1814640 1 librados: init done 2024-03-29T11:04:37.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.947+0000 7ff3d3fff640 1 -- 172.21.15.71:0/478867030 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3dc0038a0 con 0x7ff3ec0b0770 2024-03-29T11:04:37.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.947+0000 7ff3d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:37.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.947+0000 7ff3d3fff640 1 -- 172.21.15.71:0/478867030 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7ff3dc003bb0 con 0x7ff3ec0b0770 2024-03-29T11:04:37.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.947+0000 7ff3d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:37.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.947+0000 7ff3d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:04:37.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.947+0000 7ff3d3fff640 10 monclient: dump: 2024-03-29T11:04:37.936 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:37.936 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:37.936 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:37.936 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:37.936 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:37.936 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:37.936 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:37.936 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:37.936 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:37.936 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:37.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.948+0000 7ff3e8ff9640 4 set_mon_vals no callback set 2024-03-29T11:04:37.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.949+0000 7ff3f1814640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:37.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.949+0000 7ff3f1814640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:37.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.949+0000 7ff3f1814640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:04:37.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.949+0000 7ff3f1814640 1 -- 172.21.15.71:0/478867030 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3cc005080 con 0x7ff3ec0b0770 2024-03-29T11:04:37.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.952+0000 7ff3d3fff640 1 -- 172.21.15.71:0/478867030 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7ff3dc019b80 con 0x7ff3ec0b0770 2024-03-29T11:04:37.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.953+0000 7ff3d3fff640 1 -- 172.21.15.71:0/478867030 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7ff3dc083920 con 0x7ff3ec0b0770 2024-03-29T11:04:37.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.953+0000 7ff3d3fff640 10 client.4354.objecter ms_dispatch 0x7ff3ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:04:37.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.954+0000 7ff3d3fff640 3 client.4354.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:04:37.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.954+0000 7ff3d3fff640 3 client.4354.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:04:37.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.954+0000 7ff3d3fff640 20 client.4354.objecter dump_active .. 0 homeless 2024-03-29T11:04:37.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.954+0000 7ff3eaffd640 10 monclient: get_auth_request con 0x7ff3d806df40 auth_method 0 2024-03-29T11:04:37.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.956+0000 7ff3d3fff640 10 client.4354.objecter ms_handle_connect 0x7ff3d806df40 2024-03-29T11:04:37.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.960+0000 7ff3d3fff640 1 -- 172.21.15.71:0/478867030 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff3dc04f200 con 0x7ff3ec0b0770 2024-03-29T11:04:37.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.960+0000 7ff3d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:37.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:37.960+0000 7ff3d3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:04:38.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:38.287+0000 7ff3f1814640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:38.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:38.287+0000 7ff3f1814640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:38.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:38.287+0000 7ff3f1814640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:04:38.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:38.287+0000 7ff3f1814640 1 -- 172.21.15.71:0/478867030 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3cc005330 con 0x7ff3ec0b0770 2024-03-29T11:04:38.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:38.290+0000 7ff3d3fff640 1 -- 172.21.15.71:0/478867030 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7ff3dc051130 con 0x7ff3ec0b0770 2024-03-29T11:04:38.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:38.290+0000 7ff3d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:38.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:38.290+0000 7ff3d3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:04:38.271 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:04:38.271 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:04:38.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:38.294+0000 7ff3f1814640 10 librados: watch_flush enter 2024-03-29T11:04:38.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:38.294+0000 7ff3f1814640 10 librados: watch_flush exit 2024-03-29T11:04:38.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:38.294+0000 7ff3f1814640 20 client.4354.objecter shutdown clearing up homeless session... 2024-03-29T11:04:38.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:38.294+0000 7ff3f1814640 10 client.4354.objecter successfully canceled tick 2024-03-29T11:04:38.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:38.294+0000 7ff3f1814640 1 -- 172.21.15.71:0/478867030 >> v1:172.21.15.137:6816/973332665 conn(0x7ff3d806df40 legacy=0x7ff3d8070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:38.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:38.294+0000 7ff3f1814640 10 monclient: shutdown 2024-03-29T11:04:38.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:38.294+0000 7ff3f1814640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:38.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:38.294+0000 7ff3f1814640 1 -- 172.21.15.71:0/478867030 >> v1:172.21.15.137:6789/0 conn(0x7ff3ec0b0770 legacy=0x7ff3ec126dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:38.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:38.295+0000 7ff3f1814640 1 -- 172.21.15.71:0/478867030 shutdown_connections 2024-03-29T11:04:38.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:38.295+0000 7ff3f1814640 1 -- 172.21.15.71:0/478867030 >> 172.21.15.71:0/478867030 conn(0x7ff3ec06b660 msgr2=0x7ff3ec06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:38.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:38.295+0000 7ff3f1814640 1 -- 172.21.15.71:0/478867030 shutdown_connections 2024-03-29T11:04:38.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:38.295+0000 7ff3f1814640 1 -- 172.21.15.71:0/478867030 wait complete. 2024-03-29T11:04:38.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:38.295+0000 7ff3f1814640 1 librados: shutdown 2024-03-29T11:04:38.292 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:04:45.097 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:04:45.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.272+0000 7f9b58938640 10 monclient: get_monmap_and_config 2024-03-29T11:04:45.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.272+0000 7f9b58938640 10 monclient: build_initial_monmap 2024-03-29T11:04:45.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.272+0000 7f9b58938640 10 monclient: monmap: 2024-03-29T11:04:45.252 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:04:45.252 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:45.252 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:04:45.274004+0000 2024-03-29T11:04:45.252 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:04:45.274004+0000 2024-03-29T11:04:45.253 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:45.253 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:45.253 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:04:45.253 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:04:45.253 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:04:45.253 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:45.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.273+0000 7f9b58938640 1 Processor -- start 2024-03-29T11:04:45.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.273+0000 7f9b58938640 1 -- start start 2024-03-29T11:04:45.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.273+0000 7f9b58938640 10 monclient: init 2024-03-29T11:04:45.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.274+0000 7f9b58938640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:45.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.274+0000 7f9b58938640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:04:45.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.274+0000 7f9b58938640 10 monclient(hunting): picked mon.noname-a con 0x7f9b5406ade0 addr v1:172.21.15.71:6789/0 2024-03-29T11:04:45.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.274+0000 7f9b58938640 10 monclient(hunting): picked mon.noname-b con 0x7f9b541408e0 addr v1:172.21.15.71:6790/0 2024-03-29T11:04:45.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.274+0000 7f9b58938640 10 monclient(hunting): picked mon.noname-c con 0x7f9b54139f90 addr v1:172.21.15.137:6789/0 2024-03-29T11:04:45.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.274+0000 7f9b58938640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b54071210 con 0x7f9b5406ade0 2024-03-29T11:04:45.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.274+0000 7f9b58938640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b540b08e0 con 0x7f9b54139f90 2024-03-29T11:04:45.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.274+0000 7f9b58938640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b540b0bc0 con 0x7f9b541408e0 2024-03-29T11:04:45.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.274+0000 7f9b58938640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:45.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.274+0000 7f9b58938640 10 monclient(hunting): authenticate will time out at 1041.533497s 2024-03-29T11:04:45.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.277+0000 7f9b5359e640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f9b54139f90 0x7f9b5414b6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:55671/0 (socket says 172.21.15.71:55671) 2024-03-29T11:04:45.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.277+0000 7f9b5359e640 1 -- 172.21.15.71:0/1332740973 learned_addr learned my addr 172.21.15.71:0/1332740973 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:04:45.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.280+0000 7f9b51d9b640 1 -- 172.21.15.71:0/1332740973 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 91586025 0 0) 0x7f9b540b08e0 con 0x7f9b54139f90 2024-03-29T11:04:45.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.280+0000 7f9b51d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:04:45.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.280+0000 7f9b51d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:45.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.280+0000 7f9b51d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:45.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.280+0000 7f9b51d9b640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:04:45.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.280+0000 7f9b51d9b640 1 -- 172.21.15.71:0/1332740973 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b40003520 con 0x7f9b54139f90 2024-03-29T11:04:45.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.280+0000 7f9b51d9b640 1 -- 172.21.15.71:0/1332740973 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1202116133 0 0) 0x7f9b540b0bc0 con 0x7f9b541408e0 2024-03-29T11:04:45.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.280+0000 7f9b51d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:04:45.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.280+0000 7f9b51d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:45.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.280+0000 7f9b51d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:45.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.280+0000 7f9b51d9b640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:04:45.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.280+0000 7f9b51d9b640 1 -- 172.21.15.71:0/1332740973 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b540b08e0 con 0x7f9b541408e0 2024-03-29T11:04:45.262 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.281+0000 7f9b51d9b640 1 -- 172.21.15.71:0/1332740973 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3126037092 0 0) 0x7f9b40003520 con 0x7f9b54139f90 2024-03-29T11:04:45.262 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.281+0000 7f9b51d9b640 1 -- 172.21.15.71:0/1332740973 >> v1:172.21.15.71:6790/0 conn(0x7f9b541408e0 legacy=0x7f9b54139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:45.262 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.281+0000 7f9b51d9b640 1 -- 172.21.15.71:0/1332740973 >> v1:172.21.15.71:6789/0 conn(0x7f9b5406ade0 legacy=0x7f9b54070b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:04:45.262 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.281+0000 7f9b51d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:04:45.262 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.281+0000 7f9b51d9b640 1 monclient: found mon.noname-c 2024-03-29T11:04:45.262 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.281+0000 7f9b51d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:45.262 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.281+0000 7f9b51d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.137:6789/0 2024-03-29T11:04:45.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.281+0000 7f9b51d9b640 1 -- 172.21.15.71:0/1332740973 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b54070910 con 0x7f9b54139f90 2024-03-29T11:04:45.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.281+0000 7f9b51d9b640 10 monclient: _finish_auth 0 2024-03-29T11:04:45.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.281+0000 7f9b51d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:04:45.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.281+0000 7f9b51d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:45.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.281+0000 7f9b51d9b640 1 -- 172.21.15.71:0/1332740973 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9b44001a10 con 0x7f9b54139f90 2024-03-29T11:04:45.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.281+0000 7f9b51d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:45.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.281+0000 7f9b51d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:04:45.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.281+0000 7f9b51d9b640 10 monclient: dump: 2024-03-29T11:04:45.263 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:45.263 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:45.263 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:45.263 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:45.263 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:45.263 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:45.264 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:45.264 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:45.264 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:45.264 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:45.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.282+0000 7f9b58938640 5 monclient: authenticate success, global_id 4357 2024-03-29T11:04:45.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.282+0000 7f9b58938640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:04:45.266 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.285+0000 7f9b51d9b640 1 -- 172.21.15.71:0/1332740973 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b44001b70 con 0x7f9b54139f90 2024-03-29T11:04:45.266 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.285+0000 7f9b51d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:45.266 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.285+0000 7f9b51d9b640 1 -- 172.21.15.71:0/1332740973 <== mon.1 v1:172.21.15.137:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9b44001f80 con 0x7f9b54139f90 2024-03-29T11:04:45.266 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.285+0000 7f9b58938640 10 monclient: get_monmap_and_config success 2024-03-29T11:04:45.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.285+0000 7f9b58938640 4 set_mon_vals no callback set 2024-03-29T11:04:45.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.286+0000 7f9b58938640 10 monclient: shutdown 2024-03-29T11:04:45.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.286+0000 7f9b58938640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:45.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.286+0000 7f9b58938640 1 -- 172.21.15.71:0/1332740973 >> v1:172.21.15.137:6789/0 conn(0x7f9b54139f90 legacy=0x7f9b5414b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:45.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.286+0000 7f9b58938640 1 -- 172.21.15.71:0/1332740973 shutdown_connections 2024-03-29T11:04:45.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.286+0000 7f9b58938640 1 -- 172.21.15.71:0/1332740973 >> 172.21.15.71:0/1332740973 conn(0x7f9b54067420 msgr2=0x7f9b540b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:45.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.286+0000 7f9b51d9b640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:04:45.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.288+0000 7f9b58938640 1 -- 172.21.15.71:0/1332740973 shutdown_connections 2024-03-29T11:04:45.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.288+0000 7f9b58938640 1 -- 172.21.15.71:0/1332740973 wait complete. 2024-03-29T11:04:45.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.290+0000 7f9b58938640 10 monclient: build_initial_monmap 2024-03-29T11:04:45.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.290+0000 7f9b58938640 10 monclient: monmap: 2024-03-29T11:04:45.270 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:04:45.270 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:04:45.270 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:04:45.270 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:04:45.270 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:45.270 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:45.270 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:04:45.270 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:04:45.271 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:04:45.271 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:45.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.290+0000 7f9b58938640 1 librados: starting msgr at 2024-03-29T11:04:45.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.290+0000 7f9b58938640 1 librados: starting objecter 2024-03-29T11:04:45.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.291+0000 7f9b58938640 1 Processor -- start 2024-03-29T11:04:45.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.291+0000 7f9b58938640 1 -- start start 2024-03-29T11:04:45.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.291+0000 7f9b58938640 1 librados: setting wanted keys 2024-03-29T11:04:45.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.291+0000 7f9b58938640 1 librados: calling monclient init 2024-03-29T11:04:45.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.291+0000 7f9b58938640 10 monclient: init 2024-03-29T11:04:45.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.292+0000 7f9b58938640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:45.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.292+0000 7f9b58938640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:04:45.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.292+0000 7f9b58938640 10 monclient(hunting): picked mon.noname-a con 0x7f9b5406ade0 addr v1:172.21.15.71:6789/0 2024-03-29T11:04:45.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.296+0000 7f9b58938640 10 monclient(hunting): picked mon.noname-b con 0x7f9b541408e0 addr v1:172.21.15.137:6789/0 2024-03-29T11:04:45.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.298+0000 7f9b58938640 10 monclient(hunting): picked mon.noname-c con 0x7f9b541daa20 addr v1:172.21.15.71:6790/0 2024-03-29T11:04:45.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.299+0000 7f9b58938640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b54143ad0 con 0x7f9b5406ade0 2024-03-29T11:04:45.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.299+0000 7f9b58938640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b54142ac0 con 0x7f9b541408e0 2024-03-29T11:04:45.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.299+0000 7f9b58938640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b54142d30 con 0x7f9b541daa20 2024-03-29T11:04:45.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.299+0000 7f9b58938640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:45.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.299+0000 7f9b58938640 10 monclient(hunting): authenticate will time out at 1041.558677s 2024-03-29T11:04:45.282 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.299+0000 7f9b52d9d640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f9b5406ade0 0x7f9b54141100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:33445/0 (socket says 172.21.15.71:33445) 2024-03-29T11:04:45.283 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.302+0000 7f9b52d9d640 1 -- 172.21.15.71:0/3806533204 learned_addr learned my addr 172.21.15.71:0/3806533204 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:04:45.283 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.299+0000 7f9b5259c640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f9b541408e0 0x7f9b541da310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:55783/0 (socket says 172.21.15.71:55783) 2024-03-29T11:04:45.284 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.304+0000 7f9b4b7fe640 10 client.?.objecter ms_handle_connect 0x7f9b541408e0 2024-03-29T11:04:45.284 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.304+0000 7f9b4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:45.284 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.304+0000 7f9b4b7fe640 10 client.?.objecter ms_handle_connect 0x7f9b541daa20 2024-03-29T11:04:45.284 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.304+0000 7f9b4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:45.285 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.305+0000 7f9b4b7fe640 10 client.?.objecter ms_handle_connect 0x7f9b5406ade0 2024-03-29T11:04:45.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.305+0000 7f9b4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:45.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.305+0000 7f9b4b7fe640 1 -- 172.21.15.71:0/3806533204 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569709079 0 0) 0x7f9b54142ac0 con 0x7f9b541408e0 2024-03-29T11:04:45.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.305+0000 7f9b4b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:04:45.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.305+0000 7f9b4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:45.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.305+0000 7f9b4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:45.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.305+0000 7f9b4b7fe640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:04:45.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.305+0000 7f9b4b7fe640 1 -- 172.21.15.71:0/3806533204 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b34004eb0 con 0x7f9b541408e0 2024-03-29T11:04:45.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.305+0000 7f9b4b7fe640 1 -- 172.21.15.71:0/3806533204 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3327556501 0 0) 0x7f9b54143ad0 con 0x7f9b5406ade0 2024-03-29T11:04:45.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.305+0000 7f9b4b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:04:45.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.305+0000 7f9b4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:45.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.305+0000 7f9b4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:45.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.305+0000 7f9b4b7fe640 10 monclient(hunting): my global_id is 4395 2024-03-29T11:04:45.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.305+0000 7f9b4b7fe640 1 -- 172.21.15.71:0/3806533204 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b54142ac0 con 0x7f9b5406ade0 2024-03-29T11:04:45.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.307+0000 7f9b4b7fe640 1 -- 172.21.15.71:0/3806533204 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486920775 0 0) 0x7f9b54142d30 con 0x7f9b541daa20 2024-03-29T11:04:45.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.307+0000 7f9b4b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:04:45.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.307+0000 7f9b4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:45.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.307+0000 7f9b4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:45.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.307+0000 7f9b4b7fe640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:04:45.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.307+0000 7f9b4b7fe640 1 -- 172.21.15.71:0/3806533204 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b54143ad0 con 0x7f9b541daa20 2024-03-29T11:04:45.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.307+0000 7f9b4b7fe640 1 -- 172.21.15.71:0/3806533204 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2746292033 0 0) 0x7f9b54142ac0 con 0x7f9b5406ade0 2024-03-29T11:04:45.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.307+0000 7f9b4b7fe640 1 -- 172.21.15.71:0/3806533204 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b54142d30 con 0x7f9b5406ade0 2024-03-29T11:04:45.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.307+0000 7f9b4b7fe640 1 -- 172.21.15.71:0/3806533204 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9b4c003fc0 con 0x7f9b5406ade0 2024-03-29T11:04:45.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.307+0000 7f9b4b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:45.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.307+0000 7f9b4b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:04:45.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.307+0000 7f9b4b7fe640 10 monclient(hunting): dump: 2024-03-29T11:04:45.288 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:45.288 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:45.288 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:45.289 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:45.289 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:45.289 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:45.289 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:45.289 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:45.289 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:45.289 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:45.289 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.307+0000 7f9b4b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:04:45.289 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.307+0000 7f9b4b7fe640 1 -- 172.21.15.71:0/3806533204 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4015333921 0 0) 0x7f9b34004eb0 con 0x7f9b541408e0 2024-03-29T11:04:45.289 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.307+0000 7f9b4b7fe640 1 -- 172.21.15.71:0/3806533204 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b34006b30 con 0x7f9b541408e0 2024-03-29T11:04:45.289 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.308+0000 7f9b4b7fe640 1 -- 172.21.15.71:0/3806533204 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9b3c0017e0 con 0x7f9b541408e0 2024-03-29T11:04:45.289 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.308+0000 7f9b4b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:45.289 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.308+0000 7f9b4b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:04:45.289 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.308+0000 7f9b4b7fe640 10 monclient(hunting): dump: 2024-03-29T11:04:45.290 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:45.290 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:45.290 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:45.290 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:45.290 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:45.290 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:45.290 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:45.290 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:45.290 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:45.290 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:45.291 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.308+0000 7f9b4b7fe640 1 -- 172.21.15.71:0/3806533204 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 96614896 0 0) 0x7f9b54143ad0 con 0x7f9b541daa20 2024-03-29T11:04:45.291 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.308+0000 7f9b4b7fe640 1 -- 172.21.15.71:0/3806533204 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b340050f0 con 0x7f9b541daa20 2024-03-29T11:04:45.291 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.308+0000 7f9b4b7fe640 1 -- 172.21.15.71:0/3806533204 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9b44001ec0 con 0x7f9b541daa20 2024-03-29T11:04:45.291 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.308+0000 7f9b4b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:45.291 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.308+0000 7f9b4b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:04:45.291 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.308+0000 7f9b4b7fe640 10 monclient(hunting): dump: 2024-03-29T11:04:45.291 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:45.291 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:45.291 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:45.291 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:45.292 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:45.292 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:45.292 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:45.292 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:45.292 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:45.292 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:45.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.308+0000 7f9b4b7fe640 1 -- 172.21.15.71:0/3806533204 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 565782535 0 0) 0x7f9b340050f0 con 0x7f9b541daa20 2024-03-29T11:04:45.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.308+0000 7f9b4b7fe640 1 -- 172.21.15.71:0/3806533204 >> v1:172.21.15.137:6789/0 conn(0x7f9b541408e0 legacy=0x7f9b541da310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:45.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.308+0000 7f9b4b7fe640 1 -- 172.21.15.71:0/3806533204 >> v1:172.21.15.71:6789/0 conn(0x7f9b5406ade0 legacy=0x7f9b54141100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:45.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.308+0000 7f9b4b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:04:45.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.308+0000 7f9b4b7fe640 1 monclient: found mon.c 2024-03-29T11:04:45.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.308+0000 7f9b4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:45.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.308+0000 7f9b4b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:04:45.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.308+0000 7f9b4b7fe640 1 -- 172.21.15.71:0/3806533204 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b54143010 con 0x7f9b541daa20 2024-03-29T11:04:45.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.308+0000 7f9b4b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:04:45.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.308+0000 7f9b4b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:04:45.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.308+0000 7f9b4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:45.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.308+0000 7f9b58938640 5 monclient: authenticate success, global_id 4406 2024-03-29T11:04:45.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.309+0000 7f9b58938640 10 monclient: _renew_subs 2024-03-29T11:04:45.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.309+0000 7f9b58938640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:04:45.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.309+0000 7f9b58938640 1 -- 172.21.15.71:0/3806533204 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b541432f0 con 0x7f9b541daa20 2024-03-29T11:04:45.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.309+0000 7f9b58938640 10 client.4406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:04:45.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.309+0000 7f9b58938640 10 monclient: _renew_subs 2024-03-29T11:04:45.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.309+0000 7f9b58938640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:04:45.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.309+0000 7f9b58938640 1 -- 172.21.15.71:0/3806533204 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b540ae180 con 0x7f9b541daa20 2024-03-29T11:04:45.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.309+0000 7f9b58938640 1 librados: init done 2024-03-29T11:04:45.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.311+0000 7f9b497fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:45.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.311+0000 7f9b497fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:45.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.311+0000 7f9b497fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:04:45.294 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.311+0000 7f9b497fa640 1 -- 172.21.15.71:0/3806533204 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b54143810 con 0x7f9b541daa20 2024-03-29T11:04:45.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.319+0000 7f9b4b7fe640 1 -- 172.21.15.71:0/3806533204 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b44002040 con 0x7f9b541daa20 2024-03-29T11:04:45.300 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.319+0000 7f9b4b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:45.300 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.319+0000 7f9b4b7fe640 1 -- 172.21.15.71:0/3806533204 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9b44001ec0 con 0x7f9b541daa20 2024-03-29T11:04:45.300 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.319+0000 7f9b4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:45.300 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.319+0000 7f9b4b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:04:45.300 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.319+0000 7f9b4b7fe640 10 monclient: dump: 2024-03-29T11:04:45.300 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:45.300 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:45.300 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:45.300 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:45.300 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:45.300 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:45.300 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:45.300 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:45.300 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:45.301 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:45.301 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.319+0000 7f9b4b7fe640 1 -- 172.21.15.71:0/3806533204 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f9b4401ac80 con 0x7f9b541daa20 2024-03-29T11:04:45.301 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.319+0000 7f9b5159a640 4 set_mon_vals no callback set 2024-03-29T11:04:45.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.320+0000 7f9b4b7fe640 1 -- 172.21.15.71:0/3806533204 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f9b440848e0 con 0x7f9b541daa20 2024-03-29T11:04:45.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.320+0000 7f9b4b7fe640 10 client.4406.objecter ms_dispatch 0x7f9b54000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:04:45.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.320+0000 7f9b4b7fe640 3 client.4406.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:04:45.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.320+0000 7f9b4b7fe640 3 client.4406.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:04:45.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.320+0000 7f9b4b7fe640 20 client.4406.objecter dump_active .. 0 homeless 2024-03-29T11:04:45.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.321+0000 7f9b52d9d640 10 monclient: get_auth_request con 0x7f9b3406ea60 auth_method 0 2024-03-29T11:04:45.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.321+0000 7f9b4b7fe640 1 -- 172.21.15.71:0/3806533204 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9b44050070 con 0x7f9b541daa20 2024-03-29T11:04:45.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.321+0000 7f9b4b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:45.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.321+0000 7f9b4b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:04:45.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.321+0000 7f9b4b7fe640 10 client.4406.objecter ms_handle_connect 0x7f9b3406ea60 2024-03-29T11:04:45.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.683+0000 7f9b497fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:45.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.683+0000 7f9b497fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:45.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.683+0000 7f9b497fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:04:45.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.683+0000 7f9b497fa640 1 -- 172.21.15.71:0/3806533204 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b540ae430 con 0x7f9b541daa20 2024-03-29T11:04:45.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.686+0000 7f9b4b7fe640 1 -- 172.21.15.71:0/3806533204 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f9b44050070 con 0x7f9b541daa20 2024-03-29T11:04:45.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.686+0000 7f9b4b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:45.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.686+0000 7f9b4b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:04:45.668 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:04:45.668 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:04:45.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.691+0000 7f9b497fa640 10 librados: watch_flush enter 2024-03-29T11:04:45.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.691+0000 7f9b497fa640 10 librados: watch_flush exit 2024-03-29T11:04:45.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.691+0000 7f9b497fa640 20 client.4406.objecter shutdown clearing up homeless session... 2024-03-29T11:04:45.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.691+0000 7f9b497fa640 10 client.4406.objecter successfully canceled tick 2024-03-29T11:04:45.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.691+0000 7f9b497fa640 1 -- 172.21.15.71:0/3806533204 >> v1:172.21.15.137:6816/973332665 conn(0x7f9b3406ea60 legacy=0x7f9b34070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:45.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.691+0000 7f9b497fa640 10 monclient: shutdown 2024-03-29T11:04:45.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.691+0000 7f9b497fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:45.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.691+0000 7f9b497fa640 1 -- 172.21.15.71:0/3806533204 >> v1:172.21.15.71:6790/0 conn(0x7f9b541daa20 legacy=0x7f9b540ad520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:45.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.692+0000 7f9b497fa640 1 -- 172.21.15.71:0/3806533204 shutdown_connections 2024-03-29T11:04:45.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.692+0000 7f9b497fa640 1 -- 172.21.15.71:0/3806533204 >> 172.21.15.71:0/3806533204 conn(0x7f9b54067420 msgr2=0x7f9b54069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:45.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.693+0000 7f9b497fa640 1 -- 172.21.15.71:0/3806533204 shutdown_connections 2024-03-29T11:04:45.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.693+0000 7f9b497fa640 1 -- 172.21.15.71:0/3806533204 wait complete. 2024-03-29T11:04:45.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:45.693+0000 7f9b497fa640 1 librados: shutdown 2024-03-29T11:04:45.695 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:04:52.502 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:04:52.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.672+0000 7f4a63fff640 10 monclient: get_monmap_and_config 2024-03-29T11:04:52.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.672+0000 7f4a63fff640 10 monclient: build_initial_monmap 2024-03-29T11:04:52.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.672+0000 7f4a63fff640 10 monclient: monmap: 2024-03-29T11:04:52.653 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:04:52.653 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:52.653 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:04:52.673219+0000 2024-03-29T11:04:52.653 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:04:52.673219+0000 2024-03-29T11:04:52.653 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:52.653 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:52.653 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:04:52.653 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:04:52.653 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:04:52.654 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:52.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.673+0000 7f4a63fff640 1 Processor -- start 2024-03-29T11:04:52.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.673+0000 7f4a63fff640 1 -- start start 2024-03-29T11:04:52.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.673+0000 7f4a63fff640 10 monclient: init 2024-03-29T11:04:52.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.674+0000 7f4a63fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:52.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.674+0000 7f4a63fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:04:52.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.674+0000 7f4a63fff640 10 monclient(hunting): picked mon.noname-b con 0x7f4a5c006800 addr v1:172.21.15.71:6790/0 2024-03-29T11:04:52.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.674+0000 7f4a63fff640 10 monclient(hunting): picked mon.noname-c con 0x7f4a5c006be0 addr v1:172.21.15.137:6789/0 2024-03-29T11:04:52.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.674+0000 7f4a63fff640 10 monclient(hunting): picked mon.noname-a con 0x7f4a5c0324a0 addr v1:172.21.15.71:6789/0 2024-03-29T11:04:52.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.674+0000 7f4a63fff640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5c032010 con 0x7f4a5c0324a0 2024-03-29T11:04:52.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.674+0000 7f4a63fff640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5c02bc00 con 0x7f4a5c006be0 2024-03-29T11:04:52.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.674+0000 7f4a63fff640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5c032960 con 0x7f4a5c006800 2024-03-29T11:04:52.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.674+0000 7f4a63fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:52.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.674+0000 7f4a63fff640 10 monclient(hunting): authenticate will time out at 1048.932541s 2024-03-29T11:04:52.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.677+0000 7f4a627fc640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f4a5c006be0 0x7f4a5c0e6ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:57987/0 (socket says 172.21.15.71:57987) 2024-03-29T11:04:52.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.677+0000 7f4a627fc640 1 -- 172.21.15.71:0/934300196 learned_addr learned my addr 172.21.15.71:0/934300196 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:04:52.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.677+0000 7f4a61ffb640 1 -- 172.21.15.71:0/934300196 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 644860310 0 0) 0x7f4a5c032960 con 0x7f4a5c006800 2024-03-29T11:04:52.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.677+0000 7f4a61ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:04:52.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.677+0000 7f4a61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:52.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.677+0000 7f4a61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:52.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.677+0000 7f4a61ffb640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:04:52.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.677+0000 7f4a61ffb640 1 -- 172.21.15.71:0/934300196 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a4c003520 con 0x7f4a5c006800 2024-03-29T11:04:52.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.678+0000 7f4a61ffb640 1 -- 172.21.15.71:0/934300196 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1762931032 0 0) 0x7f4a4c003520 con 0x7f4a5c006800 2024-03-29T11:04:52.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.678+0000 7f4a61ffb640 1 -- 172.21.15.71:0/934300196 >> v1:172.21.15.137:6789/0 conn(0x7f4a5c006be0 legacy=0x7f4a5c0e6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:52.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.678+0000 7f4a61ffb640 1 -- 172.21.15.71:0/934300196 >> v1:172.21.15.71:6789/0 conn(0x7f4a5c0324a0 legacy=0x7f4a5c0f50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:04:52.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.678+0000 7f4a61ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:04:52.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.678+0000 7f4a61ffb640 1 monclient: found mon.noname-b 2024-03-29T11:04:52.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.678+0000 7f4a61ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:52.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.678+0000 7f4a61ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:04:52.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.678+0000 7f4a61ffb640 1 -- 172.21.15.71:0/934300196 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5c0ed440 con 0x7f4a5c006800 2024-03-29T11:04:52.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.678+0000 7f4a61ffb640 10 monclient: _finish_auth 0 2024-03-29T11:04:52.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.678+0000 7f4a61ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:04:52.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.678+0000 7f4a61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:52.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.678+0000 7f4a63fff640 5 monclient: authenticate success, global_id 4409 2024-03-29T11:04:52.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.678+0000 7f4a63fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:04:52.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.678+0000 7f4a61ffb640 1 -- 172.21.15.71:0/934300196 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4a58001690 con 0x7f4a5c006800 2024-03-29T11:04:52.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.678+0000 7f4a61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:52.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.678+0000 7f4a61ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:04:52.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.678+0000 7f4a61ffb640 10 monclient: dump: 2024-03-29T11:04:52.661 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:52.661 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:52.661 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:52.661 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:52.661 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:52.661 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:52.661 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:52.661 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:52.661 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:52.661 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:52.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.678+0000 7f4a63fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:04:52.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.678+0000 7f4a61ffb640 1 -- 172.21.15.71:0/934300196 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a58001920 con 0x7f4a5c006800 2024-03-29T11:04:52.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.679+0000 7f4a61ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:52.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.679+0000 7f4a63fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:04:52.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.679+0000 7f4a63fff640 4 set_mon_vals no callback set 2024-03-29T11:04:52.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.679+0000 7f4a63fff640 10 monclient: shutdown 2024-03-29T11:04:52.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.679+0000 7f4a63fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:52.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.679+0000 7f4a63fff640 1 -- 172.21.15.71:0/934300196 >> v1:172.21.15.71:6790/0 conn(0x7f4a5c006800 legacy=0x7f4a5c0ecd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:52.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.680+0000 7f4a63fff640 1 -- 172.21.15.71:0/934300196 shutdown_connections 2024-03-29T11:04:52.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.680+0000 7f4a63fff640 1 -- 172.21.15.71:0/934300196 >> 172.21.15.71:0/934300196 conn(0x7f4a5c0058f0 msgr2=0x7f4a5c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:52.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.680+0000 7f4a63fff640 1 -- 172.21.15.71:0/934300196 shutdown_connections 2024-03-29T11:04:52.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.680+0000 7f4a63fff640 1 -- 172.21.15.71:0/934300196 wait complete. 2024-03-29T11:04:52.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.681+0000 7f4a63fff640 10 monclient: build_initial_monmap 2024-03-29T11:04:52.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.681+0000 7f4a63fff640 10 monclient: monmap: 2024-03-29T11:04:52.664 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:04:52.664 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:04:52.664 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:04:52.664 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:04:52.664 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:52.664 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:52.664 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:04:52.664 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:04:52.664 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:04:52.664 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:52.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.682+0000 7f4a63fff640 1 librados: starting msgr at 2024-03-29T11:04:52.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.682+0000 7f4a63fff640 1 librados: starting objecter 2024-03-29T11:04:52.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.682+0000 7f4a63fff640 1 Processor -- start 2024-03-29T11:04:52.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.682+0000 7f4a63fff640 1 -- start start 2024-03-29T11:04:52.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.682+0000 7f4a63fff640 1 librados: setting wanted keys 2024-03-29T11:04:52.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.682+0000 7f4a63fff640 1 librados: calling monclient init 2024-03-29T11:04:52.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.682+0000 7f4a63fff640 10 monclient: init 2024-03-29T11:04:52.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.683+0000 7f4a63fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:52.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.683+0000 7f4a63fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:04:52.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.686+0000 7f4a63fff640 10 monclient(hunting): picked mon.noname-b con 0x7f4a5c006800 addr v1:172.21.15.137:6789/0 2024-03-29T11:04:52.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.687+0000 7f4a62ffd640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f4a5c006800 0x7f4a5c02c580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:58045/0 (socket says 172.21.15.71:58045) 2024-03-29T11:04:52.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.687+0000 7f4a62ffd640 1 -- 172.21.15.71:0/1964337836 learned_addr learned my addr 172.21.15.71:0/1964337836 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:04:52.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.687+0000 7f4a63fff640 10 monclient(hunting): picked mon.noname-a con 0x7f4a5c0324a0 addr v1:172.21.15.71:6789/0 2024-03-29T11:04:52.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.687+0000 7f4a63fff640 10 monclient(hunting): picked mon.noname-c con 0x7f4a5c0e2dd0 addr v1:172.21.15.71:6790/0 2024-03-29T11:04:52.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.688+0000 7f4a63fff640 1 -- 172.21.15.71:0/1964337836 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5c02dba0 con 0x7f4a5c0324a0 2024-03-29T11:04:52.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.688+0000 7f4a63fff640 1 -- 172.21.15.71:0/1964337836 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5c02de10 con 0x7f4a5c006800 2024-03-29T11:04:52.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.688+0000 7f4a63fff640 1 -- 172.21.15.71:0/1964337836 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5c182ab0 con 0x7f4a5c0e2dd0 2024-03-29T11:04:52.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.688+0000 7f4a63fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:52.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.688+0000 7f4a63fff640 10 monclient(hunting): authenticate will time out at 1048.946674s 2024-03-29T11:04:52.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.688+0000 7f4a4b7fe640 10 client.?.objecter ms_handle_connect 0x7f4a5c0e2dd0 2024-03-29T11:04:52.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.688+0000 7f4a4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:52.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.688+0000 7f4a4b7fe640 1 -- 172.21.15.71:0/1964337836 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122000961 0 0) 0x7f4a5c182ab0 con 0x7f4a5c0e2dd0 2024-03-29T11:04:52.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.688+0000 7f4a4b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:04:52.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.688+0000 7f4a4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:52.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.688+0000 7f4a4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:52.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.688+0000 7f4a4b7fe640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:04:52.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.688+0000 7f4a4b7fe640 1 -- 172.21.15.71:0/1964337836 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a50004eb0 con 0x7f4a5c0e2dd0 2024-03-29T11:04:52.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.690+0000 7f4a4b7fe640 1 -- 172.21.15.71:0/1964337836 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1595914630 0 0) 0x7f4a50004eb0 con 0x7f4a5c0e2dd0 2024-03-29T11:04:52.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.690+0000 7f4a4b7fe640 1 -- 172.21.15.71:0/1964337836 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a5c182ab0 con 0x7f4a5c0e2dd0 2024-03-29T11:04:52.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.690+0000 7f4a4b7fe640 1 -- 172.21.15.71:0/1964337836 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4a6404fdd0 con 0x7f4a5c0e2dd0 2024-03-29T11:04:52.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.690+0000 7f4a4b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:52.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.690+0000 7f4a4b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:04:52.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.690+0000 7f4a4b7fe640 10 monclient(hunting): dump: 2024-03-29T11:04:52.671 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:52.671 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:52.671 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:52.671 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:52.671 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:52.671 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:52.672 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:52.672 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:52.672 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:52.672 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:52.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.690+0000 7f4a4b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:04:52.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.690+0000 7f4a4b7fe640 10 client.?.objecter ms_handle_connect 0x7f4a5c0324a0 2024-03-29T11:04:52.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.690+0000 7f4a4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:52.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.691+0000 7f4a4b7fe640 10 client.?.objecter ms_handle_connect 0x7f4a5c006800 2024-03-29T11:04:52.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.691+0000 7f4a4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:52.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.691+0000 7f4a4b7fe640 1 -- 172.21.15.71:0/1964337836 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1960934032 0 0) 0x7f4a5c182ab0 con 0x7f4a5c0e2dd0 2024-03-29T11:04:52.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.691+0000 7f4a4b7fe640 1 -- 172.21.15.71:0/1964337836 >> v1:172.21.15.137:6789/0 conn(0x7f4a5c006800 legacy=0x7f4a5c02c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:52.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.691+0000 7f4a4b7fe640 1 -- 172.21.15.71:0/1964337836 >> v1:172.21.15.71:6789/0 conn(0x7f4a5c0324a0 legacy=0x7f4a5c182fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:52.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.691+0000 7f4a4b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:04:52.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.691+0000 7f4a4b7fe640 1 monclient: found mon.c 2024-03-29T11:04:52.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.691+0000 7f4a4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:52.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.691+0000 7f4a4b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:04:52.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.691+0000 7f4a4b7fe640 1 -- 172.21.15.71:0/1964337836 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5c1836e0 con 0x7f4a5c0e2dd0 2024-03-29T11:04:52.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.691+0000 7f4a4b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:04:52.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.691+0000 7f4a4b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:04:52.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.691+0000 7f4a4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:52.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.691+0000 7f4a63fff640 5 monclient: authenticate success, global_id 4412 2024-03-29T11:04:52.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.691+0000 7f4a63fff640 10 monclient: _renew_subs 2024-03-29T11:04:52.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.691+0000 7f4a63fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:04:52.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.691+0000 7f4a63fff640 1 -- 172.21.15.71:0/1964337836 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a5c0e1d70 con 0x7f4a5c0e2dd0 2024-03-29T11:04:52.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.692+0000 7f4a4b7fe640 1 -- 172.21.15.71:0/1964337836 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a6404f400 con 0x7f4a5c0e2dd0 2024-03-29T11:04:52.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.692+0000 7f4a4b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:52.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.692+0000 7f4a4b7fe640 1 -- 172.21.15.71:0/1964337836 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4a64075710 con 0x7f4a5c0e2dd0 2024-03-29T11:04:52.678 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.692+0000 7f4a4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:52.678 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.692+0000 7f4a4b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:04:52.678 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.692+0000 7f4a4b7fe640 10 monclient: dump: 2024-03-29T11:04:52.678 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:04:52.678 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:04:52.678 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:52.678 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:04:52.678 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:52.678 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:04:52.678 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:04:52.678 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:04:52.678 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:04:52.678 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:04:52.678 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.692+0000 7f4a63fff640 10 client.4412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:04:52.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.692+0000 7f4a63fff640 10 monclient: _renew_subs 2024-03-29T11:04:52.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.692+0000 7f4a63fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:04:52.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.692+0000 7f4a63fff640 1 -- 172.21.15.71:0/1964337836 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a5c0e5810 con 0x7f4a5c0e2dd0 2024-03-29T11:04:52.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.692+0000 7f4a63fff640 1 librados: init done 2024-03-29T11:04:52.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.692+0000 7f4a617fa640 4 set_mon_vals no callback set 2024-03-29T11:04:52.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.693+0000 7f4a63fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:52.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.693+0000 7f4a63fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:52.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.693+0000 7f4a63fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:04:52.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.693+0000 7f4a63fff640 1 -- 172.21.15.71:0/1964337836 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a5c1838d0 con 0x7f4a5c0e2dd0 2024-03-29T11:04:52.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.694+0000 7f4a4b7fe640 1 -- 172.21.15.71:0/1964337836 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f4a64067bf0 con 0x7f4a5c0e2dd0 2024-03-29T11:04:52.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.694+0000 7f4a4b7fe640 1 -- 172.21.15.71:0/1964337836 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f4a640ed2b0 con 0x7f4a5c0e2dd0 2024-03-29T11:04:52.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.694+0000 7f4a4b7fe640 10 client.4412.objecter ms_dispatch 0x7f4a64000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:04:52.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.694+0000 7f4a4b7fe640 3 client.4412.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:04:52.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.694+0000 7f4a4b7fe640 3 client.4412.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:04:52.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.694+0000 7f4a4b7fe640 20 client.4412.objecter dump_active .. 0 homeless 2024-03-29T11:04:52.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.695+0000 7f4a62ffd640 10 monclient: get_auth_request con 0x7f4a5006dc30 auth_method 0 2024-03-29T11:04:52.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.695+0000 7f4a4b7fe640 10 client.4412.objecter ms_handle_connect 0x7f4a5006dc30 2024-03-29T11:04:52.682 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.703+0000 7f4a4b7fe640 1 -- 172.21.15.71:0/1964337836 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4a5c1838d0 con 0x7f4a5c0e2dd0 2024-03-29T11:04:52.682 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.703+0000 7f4a4b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:52.682 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:52.703+0000 7f4a4b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:04:53.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:53.042+0000 7f4a63fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:53.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:53.042+0000 7f4a63fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:53.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:53.042+0000 7f4a63fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:04:53.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:53.042+0000 7f4a63fff640 1 -- 172.21.15.71:0/1964337836 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a5c182c40 con 0x7f4a5c0e2dd0 2024-03-29T11:04:53.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:53.044+0000 7f4a4b7fe640 1 -- 172.21.15.71:0/1964337836 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f4a640b8b90 con 0x7f4a5c0e2dd0 2024-03-29T11:04:53.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:53.044+0000 7f4a4b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:53.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:53.044+0000 7f4a4b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:04:53.024 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:04:53.024 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:04:53.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:53.048+0000 7f4a63fff640 10 librados: watch_flush enter 2024-03-29T11:04:53.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:53.049+0000 7f4a63fff640 10 librados: watch_flush exit 2024-03-29T11:04:53.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:53.049+0000 7f4a63fff640 20 client.4412.objecter shutdown clearing up homeless session... 2024-03-29T11:04:53.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:53.049+0000 7f4a63fff640 10 client.4412.objecter successfully canceled tick 2024-03-29T11:04:53.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:53.049+0000 7f4a63fff640 1 -- 172.21.15.71:0/1964337836 >> v1:172.21.15.137:6816/973332665 conn(0x7f4a5006dc30 legacy=0x7f4a500700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:53.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:53.049+0000 7f4a63fff640 10 monclient: shutdown 2024-03-29T11:04:53.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:53.049+0000 7f4a63fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:53.029 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:53.049+0000 7f4a63fff640 1 -- 172.21.15.71:0/1964337836 >> v1:172.21.15.71:6790/0 conn(0x7f4a5c0e2dd0 legacy=0x7f4a5c0e1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:53.029 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:53.049+0000 7f4a63fff640 1 -- 172.21.15.71:0/1964337836 shutdown_connections 2024-03-29T11:04:53.029 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:53.049+0000 7f4a63fff640 1 -- 172.21.15.71:0/1964337836 >> 172.21.15.71:0/1964337836 conn(0x7f4a5c0058f0 msgr2=0x7f4a5c036a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:53.029 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:53.049+0000 7f4a63fff640 1 -- 172.21.15.71:0/1964337836 shutdown_connections 2024-03-29T11:04:53.029 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:53.049+0000 7f4a63fff640 1 -- 172.21.15.71:0/1964337836 wait complete. 2024-03-29T11:04:53.029 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:04:53.049+0000 7f4a63fff640 1 librados: shutdown 2024-03-29T11:04:53.047 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:04:59.855 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:05:00.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.024+0000 7f92f56e2640 10 monclient: get_monmap_and_config 2024-03-29T11:05:00.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.024+0000 7f92f56e2640 10 monclient: build_initial_monmap 2024-03-29T11:05:00.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.024+0000 7f92f56e2640 10 monclient: monmap: 2024-03-29T11:05:00.004 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:05:00.004 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:00.004 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:05:00.025143+0000 2024-03-29T11:05:00.004 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:05:00.025143+0000 2024-03-29T11:05:00.004 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:00.004 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:00.004 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:05:00.004 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:05:00.004 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:05:00.004 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:00.005 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.025+0000 7f92f56e2640 1 Processor -- start 2024-03-29T11:05:00.005 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.025+0000 7f92f56e2640 1 -- start start 2024-03-29T11:05:00.005 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.025+0000 7f92f56e2640 10 monclient: init 2024-03-29T11:05:00.005 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.026+0000 7f92f56e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:00.005 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.026+0000 7f92f56e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:05:00.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.026+0000 7f92f56e2640 10 monclient(hunting): picked mon.noname-c con 0x7f92f01457f0 addr v1:172.21.15.137:6789/0 2024-03-29T11:05:00.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.026+0000 7f92f56e2640 10 monclient(hunting): picked mon.noname-b con 0x7f92f013f650 addr v1:172.21.15.71:6790/0 2024-03-29T11:05:00.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.026+0000 7f92f56e2640 10 monclient(hunting): picked mon.noname-a con 0x7f92f00a9d70 addr v1:172.21.15.71:6789/0 2024-03-29T11:05:00.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.026+0000 7f92f56e2640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f00aaef0 con 0x7f92f00a9d70 2024-03-29T11:05:00.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.026+0000 7f92f56e2640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f0145bd0 con 0x7f92f01457f0 2024-03-29T11:05:00.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.026+0000 7f92f56e2640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f0145eb0 con 0x7f92f013f650 2024-03-29T11:05:00.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.026+0000 7f92f56e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:00.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.026+0000 7f92f56e2640 10 monclient(hunting): authenticate will time out at 1056.284597s 2024-03-29T11:05:00.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.026+0000 7f92ee7fc640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f92f013f650 0x7f92f00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:52570/0 (socket says 172.21.15.71:52570) 2024-03-29T11:05:00.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.026+0000 7f92ee7fc640 1 -- 172.21.15.71:0/559704771 learned_addr learned my addr 172.21.15.71:0/559704771 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:05:00.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.027+0000 7f92edffb640 1 -- 172.21.15.71:0/559704771 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2349616313 0 0) 0x7f92f0145eb0 con 0x7f92f013f650 2024-03-29T11:05:00.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.027+0000 7f92edffb640 10 monclient(hunting): _negotiate 2024-03-29T11:05:00.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.027+0000 7f92edffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:00.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.027+0000 7f92edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:00.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.027+0000 7f92edffb640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:05:00.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.027+0000 7f92edffb640 1 -- 172.21.15.71:0/559704771 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92dc003520 con 0x7f92f013f650 2024-03-29T11:05:00.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.027+0000 7f92edffb640 1 -- 172.21.15.71:0/559704771 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266043986 0 0) 0x7f92f0145bd0 con 0x7f92f01457f0 2024-03-29T11:05:00.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.027+0000 7f92edffb640 10 monclient(hunting): _negotiate 2024-03-29T11:05:00.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.027+0000 7f92edffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:00.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.027+0000 7f92edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:00.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.027+0000 7f92edffb640 10 monclient(hunting): my global_id is 4366 2024-03-29T11:05:00.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.027+0000 7f92edffb640 1 -- 172.21.15.71:0/559704771 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92f0145eb0 con 0x7f92f01457f0 2024-03-29T11:05:00.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.027+0000 7f92edffb640 1 -- 172.21.15.71:0/559704771 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1488808391 0 0) 0x7f92f00aaef0 con 0x7f92f00a9d70 2024-03-29T11:05:00.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.027+0000 7f92edffb640 10 monclient(hunting): _negotiate 2024-03-29T11:05:00.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.027+0000 7f92edffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:00.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.027+0000 7f92edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:00.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.027+0000 7f92edffb640 10 monclient(hunting): my global_id is 4401 2024-03-29T11:05:00.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.027+0000 7f92edffb640 1 -- 172.21.15.71:0/559704771 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92f0145bd0 con 0x7f92f00a9d70 2024-03-29T11:05:00.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.028+0000 7f92edffb640 1 -- 172.21.15.71:0/559704771 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2251766190 0 0) 0x7f92dc003520 con 0x7f92f013f650 2024-03-29T11:05:00.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.028+0000 7f92edffb640 1 -- 172.21.15.71:0/559704771 >> v1:172.21.15.137:6789/0 conn(0x7f92f01457f0 legacy=0x7f92f013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:00.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.028+0000 7f92edffb640 1 -- 172.21.15.71:0/559704771 >> v1:172.21.15.71:6789/0 conn(0x7f92f00a9d70 legacy=0x7f92f014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:00.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.028+0000 7f92edffb640 10 monclient: _finish_hunting 0 2024-03-29T11:05:00.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.028+0000 7f92edffb640 1 monclient: found mon.noname-b 2024-03-29T11:05:00.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.028+0000 7f92edffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:00.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.028+0000 7f92edffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:05:00.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.028+0000 7f92edffb640 1 -- 172.21.15.71:0/559704771 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f0146340 con 0x7f92f013f650 2024-03-29T11:05:00.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.028+0000 7f92edffb640 10 monclient: _finish_auth 0 2024-03-29T11:05:00.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.028+0000 7f92edffb640 10 monclient: _check_auth_tickets 2024-03-29T11:05:00.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.028+0000 7f92edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:00.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.028+0000 7f92f56e2640 5 monclient: authenticate success, global_id 4415 2024-03-29T11:05:00.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.028+0000 7f92f56e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:05:00.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.028+0000 7f92edffb640 1 -- 172.21.15.71:0/559704771 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f92e0001690 con 0x7f92f013f650 2024-03-29T11:05:00.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.028+0000 7f92edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:00.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.028+0000 7f92edffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:05:00.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.028+0000 7f92edffb640 10 monclient: dump: 2024-03-29T11:05:00.012 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:00.012 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:00.013 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:00.013 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:00.013 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:00.013 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:00.013 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:00.013 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:00.013 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:00.013 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:00.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.028+0000 7f92f56e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:05:00.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.028+0000 7f92edffb640 1 -- 172.21.15.71:0/559704771 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92e0001870 con 0x7f92f013f650 2024-03-29T11:05:00.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.028+0000 7f92edffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:00.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.028+0000 7f92f56e2640 10 monclient: get_monmap_and_config success 2024-03-29T11:05:00.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.028+0000 7f92f56e2640 4 set_mon_vals no callback set 2024-03-29T11:05:00.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.028+0000 7f92edffb640 1 -- 172.21.15.71:0/559704771 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f92e0001cc0 con 0x7f92f013f650 2024-03-29T11:05:00.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.028+0000 7f92edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:00.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.028+0000 7f92edffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:05:00.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.028+0000 7f92edffb640 10 monclient: dump: 2024-03-29T11:05:00.014 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:00.014 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:00.014 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:00.014 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:00.014 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:00.014 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:00.014 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:00.014 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:00.014 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:00.014 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:00.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.029+0000 7f92f56e2640 10 monclient: shutdown 2024-03-29T11:05:00.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.029+0000 7f92f56e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:00.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.029+0000 7f92f56e2640 1 -- 172.21.15.71:0/559704771 >> v1:172.21.15.71:6790/0 conn(0x7f92f013f650 legacy=0x7f92f00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:00.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.029+0000 7f92f56e2640 1 -- 172.21.15.71:0/559704771 shutdown_connections 2024-03-29T11:05:00.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.029+0000 7f92f56e2640 1 -- 172.21.15.71:0/559704771 >> 172.21.15.71:0/559704771 conn(0x7f92f0066a00 msgr2=0x7f92f0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:00.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.029+0000 7f92f56e2640 1 -- 172.21.15.71:0/559704771 shutdown_connections 2024-03-29T11:05:00.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.029+0000 7f92f56e2640 1 -- 172.21.15.71:0/559704771 wait complete. 2024-03-29T11:05:00.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.030+0000 7f92f56e2640 10 monclient: build_initial_monmap 2024-03-29T11:05:00.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.030+0000 7f92f56e2640 10 monclient: monmap: 2024-03-29T11:05:00.015 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:05:00.015 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:05:00.015 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:05:00.015 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:05:00.015 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:00.015 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:00.016 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:05:00.016 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:05:00.016 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:05:00.016 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:00.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.031+0000 7f92f56e2640 1 librados: starting msgr at 2024-03-29T11:05:00.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.031+0000 7f92f56e2640 1 librados: starting objecter 2024-03-29T11:05:00.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.031+0000 7f92f56e2640 1 Processor -- start 2024-03-29T11:05:00.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.031+0000 7f92f56e2640 1 -- start start 2024-03-29T11:05:00.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.031+0000 7f92f56e2640 1 librados: setting wanted keys 2024-03-29T11:05:00.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.031+0000 7f92f56e2640 1 librados: calling monclient init 2024-03-29T11:05:00.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.031+0000 7f92f56e2640 10 monclient: init 2024-03-29T11:05:00.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.032+0000 7f92f56e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:00.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.032+0000 7f92f56e2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:05:00.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.032+0000 7f92f56e2640 10 monclient(hunting): picked mon.noname-b con 0x7f92f00a9d70 addr v1:172.21.15.137:6789/0 2024-03-29T11:05:00.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.032+0000 7f92f56e2640 10 monclient(hunting): picked mon.noname-a con 0x7f92f013f650 addr v1:172.21.15.71:6789/0 2024-03-29T11:05:00.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.032+0000 7f92f56e2640 10 monclient(hunting): picked mon.noname-c con 0x7f92f01457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:05:00.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.032+0000 7f92f56e2640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f01452e0 con 0x7f92f013f650 2024-03-29T11:05:00.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.032+0000 7f92f56e2640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f0140f00 con 0x7f92f00a9d70 2024-03-29T11:05:00.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.032+0000 7f92f56e2640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f013ff30 con 0x7f92f01457f0 2024-03-29T11:05:00.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.032+0000 7f92f56e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:00.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.032+0000 7f92f56e2640 10 monclient(hunting): authenticate will time out at 1056.290437s 2024-03-29T11:05:00.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.032+0000 7f92ee7fc640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f92f013f650 0x7f92f0143a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:41050/0 (socket says 172.21.15.71:41050) 2024-03-29T11:05:00.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.032+0000 7f92ee7fc640 1 -- 172.21.15.71:0/2678079148 learned_addr learned my addr 172.21.15.71:0/2678079148 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:05:00.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.032+0000 7f92e77fe640 10 client.?.objecter ms_handle_connect 0x7f92f013f650 2024-03-29T11:05:00.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.032+0000 7f92e77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:00.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.032+0000 7f92e77fe640 10 client.?.objecter ms_handle_connect 0x7f92f00a9d70 2024-03-29T11:05:00.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.032+0000 7f92e77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:00.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.033+0000 7f92e77fe640 1 -- 172.21.15.71:0/2678079148 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934373206 0 0) 0x7f92f01452e0 con 0x7f92f013f650 2024-03-29T11:05:00.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.033+0000 7f92e77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:05:00.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.033+0000 7f92e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:00.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.033+0000 7f92e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:00.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.033+0000 7f92e77fe640 10 monclient(hunting): my global_id is 4404 2024-03-29T11:05:00.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.033+0000 7f92e77fe640 1 -- 172.21.15.71:0/2678079148 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d0004eb0 con 0x7f92f013f650 2024-03-29T11:05:00.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.033+0000 7f92e77fe640 10 client.?.objecter ms_handle_connect 0x7f92f01457f0 2024-03-29T11:05:00.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.033+0000 7f92e77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:00.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.033+0000 7f92e77fe640 1 -- 172.21.15.71:0/2678079148 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1251443950 0 0) 0x7f92d0004eb0 con 0x7f92f013f650 2024-03-29T11:05:00.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.033+0000 7f92e77fe640 1 -- 172.21.15.71:0/2678079148 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92d0005300 con 0x7f92f013f650 2024-03-29T11:05:00.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.033+0000 7f92e77fe640 1 -- 172.21.15.71:0/2678079148 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f92e0001b80 con 0x7f92f013f650 2024-03-29T11:05:00.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.033+0000 7f92e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:00.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.033+0000 7f92e77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:05:00.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.033+0000 7f92e77fe640 10 monclient(hunting): dump: 2024-03-29T11:05:00.022 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:00.022 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:00.022 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:00.022 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:00.022 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:00.022 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:00.022 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:00.022 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:00.022 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:00.022 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:00.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.033+0000 7f92e77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:05:00.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.033+0000 7f92e77fe640 1 -- 172.21.15.71:0/2678079148 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979214705 0 0) 0x7f92f013ff30 con 0x7f92f01457f0 2024-03-29T11:05:00.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.033+0000 7f92e77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:05:00.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.033+0000 7f92e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:00.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.033+0000 7f92e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:00.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.033+0000 7f92e77fe640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:05:00.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.033+0000 7f92e77fe640 1 -- 172.21.15.71:0/2678079148 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d0004eb0 con 0x7f92f01457f0 2024-03-29T11:05:00.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.033+0000 7f92e77fe640 1 -- 172.21.15.71:0/2678079148 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3681598176 0 0) 0x7f92f0140f00 con 0x7f92f00a9d70 2024-03-29T11:05:00.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.033+0000 7f92e77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:05:00.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.033+0000 7f92e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:00.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.033+0000 7f92e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:00.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.033+0000 7f92e77fe640 10 monclient(hunting): my global_id is 4369 2024-03-29T11:05:00.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.033+0000 7f92e77fe640 1 -- 172.21.15.71:0/2678079148 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92f013ff30 con 0x7f92f00a9d70 2024-03-29T11:05:00.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.034+0000 7f92e77fe640 1 -- 172.21.15.71:0/2678079148 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 656469718 0 0) 0x7f92d0005300 con 0x7f92f013f650 2024-03-29T11:05:00.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.034+0000 7f92e77fe640 1 -- 172.21.15.71:0/2678079148 >> v1:172.21.15.71:6790/0 conn(0x7f92f01457f0 legacy=0x7f92f01e6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:00.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.034+0000 7f92e77fe640 1 -- 172.21.15.71:0/2678079148 >> v1:172.21.15.137:6789/0 conn(0x7f92f00a9d70 legacy=0x7f92f0143340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:00.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.034+0000 7f92e77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:05:00.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.034+0000 7f92e77fe640 1 monclient: found mon.a 2024-03-29T11:05:00.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.034+0000 7f92e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:00.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.034+0000 7f92e77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:05:00.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.034+0000 7f92e77fe640 1 -- 172.21.15.71:0/2678079148 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f01400c0 con 0x7f92f013f650 2024-03-29T11:05:00.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.034+0000 7f92e77fe640 10 monclient: _finish_auth 0 2024-03-29T11:05:00.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.034+0000 7f92e77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:05:00.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.034+0000 7f92e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:00.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.034+0000 7f92f56e2640 5 monclient: authenticate success, global_id 4404 2024-03-29T11:05:00.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.034+0000 7f92f56e2640 10 monclient: _renew_subs 2024-03-29T11:05:00.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.034+0000 7f92f56e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:05:00.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.034+0000 7f92f56e2640 1 -- 172.21.15.71:0/2678079148 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f0144ca0 con 0x7f92f013f650 2024-03-29T11:05:00.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.034+0000 7f92f56e2640 10 client.4404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:05:00.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.034+0000 7f92f56e2640 10 monclient: _renew_subs 2024-03-29T11:05:00.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.034+0000 7f92f56e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:05:00.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.034+0000 7f92f56e2640 1 -- 172.21.15.71:0/2678079148 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f013d830 con 0x7f92f013f650 2024-03-29T11:05:00.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.034+0000 7f92f56e2640 1 librados: init done 2024-03-29T11:05:00.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.035+0000 7f92f56e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:00.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.035+0000 7f92e77fe640 1 -- 172.21.15.71:0/2678079148 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92e0004bf0 con 0x7f92f013f650 2024-03-29T11:05:00.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.035+0000 7f92e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:00.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.035+0000 7f92e77fe640 1 -- 172.21.15.71:0/2678079148 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f92e0004f00 con 0x7f92f013f650 2024-03-29T11:05:00.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.035+0000 7f92e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:00.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.035+0000 7f92e77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:05:00.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.035+0000 7f92e77fe640 10 monclient: dump: 2024-03-29T11:05:00.025 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:00.025 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:00.025 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:00.025 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:00.025 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:00.025 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:00.026 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:00.026 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:00.026 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:00.026 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:00.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.035+0000 7f92ed7fa640 4 set_mon_vals no callback set 2024-03-29T11:05:00.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.036+0000 7f92f56e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:00.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.036+0000 7f92f56e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:05:00.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.036+0000 7f92e77fe640 1 -- 172.21.15.71:0/2678079148 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f92e001aed0 con 0x7f92f013f650 2024-03-29T11:05:00.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.036+0000 7f92f56e2640 1 -- 172.21.15.71:0/2678079148 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92f0075f10 con 0x7f92f013f650 2024-03-29T11:05:00.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.037+0000 7f92e77fe640 1 -- 172.21.15.71:0/2678079148 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f92e0084b70 con 0x7f92f013f650 2024-03-29T11:05:00.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.037+0000 7f92e77fe640 10 client.4404.objecter ms_dispatch 0x7f92f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:05:00.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.037+0000 7f92e77fe640 3 client.4404.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:05:00.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.037+0000 7f92e77fe640 3 client.4404.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:05:00.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.037+0000 7f92e77fe640 20 client.4404.objecter dump_active .. 0 homeless 2024-03-29T11:05:00.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.037+0000 7f92eeffd640 10 monclient: get_auth_request con 0x7f92d006e040 auth_method 0 2024-03-29T11:05:00.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.038+0000 7f92e77fe640 10 client.4404.objecter ms_handle_connect 0x7f92d006e040 2024-03-29T11:05:00.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.045+0000 7f92e77fe640 1 -- 172.21.15.71:0/2678079148 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f92e00503c0 con 0x7f92f013f650 2024-03-29T11:05:00.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.045+0000 7f92e77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:00.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.045+0000 7f92e77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:05:00.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.413+0000 7f92f56e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:00.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.413+0000 7f92f56e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:00.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.413+0000 7f92f56e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:05:00.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.413+0000 7f92f56e2640 1 -- 172.21.15.71:0/2678079148 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92f00b3680 con 0x7f92f013f650 2024-03-29T11:05:00.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.414+0000 7f92e77fe640 1 -- 172.21.15.71:0/2678079148 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f92e00503c0 con 0x7f92f013f650 2024-03-29T11:05:00.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.414+0000 7f92e77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:00.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.414+0000 7f92e77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:05:00.393 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:05:00.393 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:05:00.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.417+0000 7f92f56e2640 10 librados: watch_flush enter 2024-03-29T11:05:00.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.417+0000 7f92f56e2640 10 librados: watch_flush exit 2024-03-29T11:05:00.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.417+0000 7f92f56e2640 20 client.4404.objecter shutdown clearing up homeless session... 2024-03-29T11:05:00.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.417+0000 7f92f56e2640 10 client.4404.objecter successfully canceled tick 2024-03-29T11:05:00.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.417+0000 7f92f56e2640 1 -- 172.21.15.71:0/2678079148 >> v1:172.21.15.137:6816/973332665 conn(0x7f92d006e040 legacy=0x7f92d0070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:00.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.417+0000 7f92f56e2640 10 monclient: shutdown 2024-03-29T11:05:00.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.417+0000 7f92f56e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:00.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.417+0000 7f92f56e2640 1 -- 172.21.15.71:0/2678079148 >> v1:172.21.15.71:6789/0 conn(0x7f92f013f650 legacy=0x7f92f0143a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:00.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.418+0000 7f92f56e2640 1 -- 172.21.15.71:0/2678079148 shutdown_connections 2024-03-29T11:05:00.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.418+0000 7f92f56e2640 1 -- 172.21.15.71:0/2678079148 >> 172.21.15.71:0/2678079148 conn(0x7f92f0066a00 msgr2=0x7f92f00b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:00.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.418+0000 7f92f56e2640 1 -- 172.21.15.71:0/2678079148 shutdown_connections 2024-03-29T11:05:00.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.418+0000 7f92f56e2640 1 -- 172.21.15.71:0/2678079148 wait complete. 2024-03-29T11:05:00.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:00.418+0000 7f92f56e2640 1 librados: shutdown 2024-03-29T11:05:00.413 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:05:07.220 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:05:07.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.383+0000 7f52f65f6640 10 monclient: get_monmap_and_config 2024-03-29T11:05:07.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.383+0000 7f52f65f6640 10 monclient: build_initial_monmap 2024-03-29T11:05:07.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.383+0000 7f52f65f6640 10 monclient: monmap: 2024-03-29T11:05:07.363 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:05:07.363 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:07.363 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:05:07.384204+0000 2024-03-29T11:05:07.363 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:05:07.384204+0000 2024-03-29T11:05:07.363 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:07.363 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:07.363 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:05:07.364 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:05:07.364 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:05:07.364 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:07.364 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.384+0000 7f52f65f6640 1 Processor -- start 2024-03-29T11:05:07.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.384+0000 7f52f65f6640 1 -- start start 2024-03-29T11:05:07.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.384+0000 7f52f65f6640 10 monclient: init 2024-03-29T11:05:07.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.385+0000 7f52f65f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:07.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.385+0000 7f52f65f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:05:07.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.385+0000 7f52f65f6640 10 monclient(hunting): picked mon.noname-b con 0x7f52f00b2800 addr v1:172.21.15.71:6790/0 2024-03-29T11:05:07.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.385+0000 7f52f65f6640 10 monclient(hunting): picked mon.noname-a con 0x7f52f00a9c00 addr v1:172.21.15.71:6789/0 2024-03-29T11:05:07.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.385+0000 7f52f65f6640 10 monclient(hunting): picked mon.noname-c con 0x7f52f00b38c0 addr v1:172.21.15.137:6789/0 2024-03-29T11:05:07.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.385+0000 7f52f65f6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f0141eb0 con 0x7f52f00a9c00 2024-03-29T11:05:07.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.385+0000 7f52f65f6640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f00b22d0 con 0x7f52f00b38c0 2024-03-29T11:05:07.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.385+0000 7f52f65f6640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f0142d30 con 0x7f52f00b2800 2024-03-29T11:05:07.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.385+0000 7f52f65f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:07.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.385+0000 7f52f65f6640 10 monclient(hunting): authenticate will time out at 1063.643628s 2024-03-29T11:05:07.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.385+0000 7f52f5df5640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f52f00b38c0 0x7f52f014d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:41712/0 (socket says 172.21.15.71:41712) 2024-03-29T11:05:07.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.385+0000 7f52f5df5640 1 -- 172.21.15.71:0/423635140 learned_addr learned my addr 172.21.15.71:0/423635140 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:05:07.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 1 -- 172.21.15.71:0/423635140 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3749014348 0 0) 0x7f52f00b22d0 con 0x7f52f00b38c0 2024-03-29T11:05:07.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 10 monclient(hunting): _negotiate 2024-03-29T11:05:07.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:07.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:07.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 10 monclient(hunting): my global_id is 4372 2024-03-29T11:05:07.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 1 -- 172.21.15.71:0/423635140 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52d8003520 con 0x7f52f00b38c0 2024-03-29T11:05:07.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 1 -- 172.21.15.71:0/423635140 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957905144 0 0) 0x7f52f0142d30 con 0x7f52f00b2800 2024-03-29T11:05:07.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 10 monclient(hunting): _negotiate 2024-03-29T11:05:07.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:07.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:07.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:05:07.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 1 -- 172.21.15.71:0/423635140 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52d8003840 con 0x7f52f00b2800 2024-03-29T11:05:07.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 1 -- 172.21.15.71:0/423635140 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3931254592 0 0) 0x7f52f0141eb0 con 0x7f52f00a9c00 2024-03-29T11:05:07.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 10 monclient(hunting): _negotiate 2024-03-29T11:05:07.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:07.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:07.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 10 monclient(hunting): my global_id is 4407 2024-03-29T11:05:07.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 1 -- 172.21.15.71:0/423635140 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52f0142d30 con 0x7f52f00a9c00 2024-03-29T11:05:07.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 1 -- 172.21.15.71:0/423635140 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3047496705 0 0) 0x7f52d8003520 con 0x7f52f00b38c0 2024-03-29T11:05:07.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 1 -- 172.21.15.71:0/423635140 >> v1:172.21.15.71:6790/0 conn(0x7f52f00b2800 legacy=0x7f52f0142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:07.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 1 -- 172.21.15.71:0/423635140 >> v1:172.21.15.71:6789/0 conn(0x7f52f00a9c00 legacy=0x7f52f013bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:07.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 10 monclient: _finish_hunting 0 2024-03-29T11:05:07.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 1 monclient: found mon.noname-c 2024-03-29T11:05:07.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:07.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.137:6789/0 2024-03-29T11:05:07.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 1 -- 172.21.15.71:0/423635140 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f00aaf90 con 0x7f52f00b38c0 2024-03-29T11:05:07.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 10 monclient: _finish_auth 0 2024-03-29T11:05:07.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 10 monclient: _check_auth_tickets 2024-03-29T11:05:07.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:07.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 1 -- 172.21.15.71:0/423635140 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f52e0001670 con 0x7f52f00b38c0 2024-03-29T11:05:07.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52f65f6640 5 monclient: authenticate success, global_id 4372 2024-03-29T11:05:07.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:07.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:05:07.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52effff640 10 monclient: dump: 2024-03-29T11:05:07.370 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:07.371 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:07.371 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:07.371 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:07.371 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:07.371 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:07.371 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:07.371 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:07.371 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:07.371 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:07.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.386+0000 7f52f65f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:05:07.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.387+0000 7f52effff640 1 -- 172.21.15.71:0/423635140 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52e00017d0 con 0x7f52f00b38c0 2024-03-29T11:05:07.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.387+0000 7f52effff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:07.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.387+0000 7f52effff640 1 -- 172.21.15.71:0/423635140 <== mon.1 v1:172.21.15.137:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f52e0001b60 con 0x7f52f00b38c0 2024-03-29T11:05:07.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.387+0000 7f52effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:07.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.387+0000 7f52effff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:05:07.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.387+0000 7f52effff640 10 monclient: dump: 2024-03-29T11:05:07.372 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:07.372 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:07.372 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:07.372 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:07.372 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:07.372 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:07.372 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:07.373 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:07.373 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:07.373 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:07.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.387+0000 7f52f65f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:05:07.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.387+0000 7f52f65f6640 4 set_mon_vals no callback set 2024-03-29T11:05:07.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.388+0000 7f52f65f6640 10 monclient: shutdown 2024-03-29T11:05:07.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.388+0000 7f52f65f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:07.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.388+0000 7f52f65f6640 1 -- 172.21.15.71:0/423635140 >> v1:172.21.15.137:6789/0 conn(0x7f52f00b38c0 legacy=0x7f52f014d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:07.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.388+0000 7f52f65f6640 1 -- 172.21.15.71:0/423635140 shutdown_connections 2024-03-29T11:05:07.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.388+0000 7f52f65f6640 1 -- 172.21.15.71:0/423635140 >> 172.21.15.71:0/423635140 conn(0x7f52f0066a00 msgr2=0x7f52f0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:07.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.388+0000 7f52f65f6640 1 -- 172.21.15.71:0/423635140 shutdown_connections 2024-03-29T11:05:07.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.388+0000 7f52f65f6640 1 -- 172.21.15.71:0/423635140 wait complete. 2024-03-29T11:05:07.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.389+0000 7f52f65f6640 10 monclient: build_initial_monmap 2024-03-29T11:05:07.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.389+0000 7f52f65f6640 10 monclient: monmap: 2024-03-29T11:05:07.380 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:05:07.380 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:05:07.380 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:05:07.380 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:05:07.380 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:07.380 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:07.380 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:05:07.380 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:05:07.381 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:05:07.381 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:07.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.390+0000 7f52f65f6640 1 librados: starting msgr at 2024-03-29T11:05:07.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.390+0000 7f52f65f6640 1 librados: starting objecter 2024-03-29T11:05:07.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.390+0000 7f52f65f6640 1 Processor -- start 2024-03-29T11:05:07.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.390+0000 7f52f65f6640 1 -- start start 2024-03-29T11:05:07.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.390+0000 7f52f65f6640 1 librados: setting wanted keys 2024-03-29T11:05:07.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.390+0000 7f52f65f6640 1 librados: calling monclient init 2024-03-29T11:05:07.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.390+0000 7f52f65f6640 10 monclient: init 2024-03-29T11:05:07.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.391+0000 7f52f65f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:07.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.391+0000 7f52f65f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:05:07.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.391+0000 7f52f65f6640 10 monclient(hunting): picked mon.noname-c con 0x7f52f00a9c00 addr v1:172.21.15.71:6790/0 2024-03-29T11:05:07.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.391+0000 7f52f65f6640 10 monclient(hunting): picked mon.noname-b con 0x7f52f00b2800 addr v1:172.21.15.137:6789/0 2024-03-29T11:05:07.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.391+0000 7f52f65f6640 10 monclient(hunting): picked mon.noname-a con 0x7f52f00b38c0 addr v1:172.21.15.71:6789/0 2024-03-29T11:05:07.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.391+0000 7f52f65f6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f00ae530 con 0x7f52f00b38c0 2024-03-29T11:05:07.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.391+0000 7f52f65f6640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f01da670 con 0x7f52f00b2800 2024-03-29T11:05:07.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.391+0000 7f52f65f6640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f01d70f0 con 0x7f52f00a9c00 2024-03-29T11:05:07.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.391+0000 7f52f65f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:07.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.391+0000 7f52f65f6640 10 monclient(hunting): authenticate will time out at 1063.650044s 2024-03-29T11:05:07.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.392+0000 7f52f55f4640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f52f00a9c00 0x7f52f00ade20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:37262/0 (socket says 172.21.15.71:37262) 2024-03-29T11:05:07.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.392+0000 7f52f55f4640 1 -- 172.21.15.71:0/1449146447 learned_addr learned my addr 172.21.15.71:0/1449146447 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:05:07.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.392+0000 7f52edffb640 10 client.?.objecter ms_handle_connect 0x7f52f00a9c00 2024-03-29T11:05:07.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.392+0000 7f52edffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:07.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.392+0000 7f52edffb640 10 client.?.objecter ms_handle_connect 0x7f52f00b38c0 2024-03-29T11:05:07.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.392+0000 7f52edffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:07.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.393+0000 7f52edffb640 10 client.?.objecter ms_handle_connect 0x7f52f00b2800 2024-03-29T11:05:07.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.393+0000 7f52edffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:07.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.393+0000 7f52edffb640 1 -- 172.21.15.71:0/1449146447 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106161926 0 0) 0x7f52f01d70f0 con 0x7f52f00a9c00 2024-03-29T11:05:07.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.393+0000 7f52edffb640 10 monclient(hunting): _negotiate 2024-03-29T11:05:07.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.393+0000 7f52edffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:07.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.393+0000 7f52edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:07.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.393+0000 7f52edffb640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:05:07.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.393+0000 7f52edffb640 1 -- 172.21.15.71:0/1449146447 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52dc004eb0 con 0x7f52f00a9c00 2024-03-29T11:05:07.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.393+0000 7f52edffb640 1 -- 172.21.15.71:0/1449146447 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1161846862 0 0) 0x7f52f00ae530 con 0x7f52f00b38c0 2024-03-29T11:05:07.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.393+0000 7f52edffb640 10 monclient(hunting): _negotiate 2024-03-29T11:05:07.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.393+0000 7f52edffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:07.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.393+0000 7f52edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:07.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.393+0000 7f52edffb640 10 monclient(hunting): my global_id is 4410 2024-03-29T11:05:07.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.393+0000 7f52edffb640 1 -- 172.21.15.71:0/1449146447 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52f01d70f0 con 0x7f52f00b38c0 2024-03-29T11:05:07.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.393+0000 7f52edffb640 1 -- 172.21.15.71:0/1449146447 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 801407840 0 0) 0x7f52f01da670 con 0x7f52f00b2800 2024-03-29T11:05:07.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.393+0000 7f52edffb640 10 monclient(hunting): _negotiate 2024-03-29T11:05:07.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.393+0000 7f52edffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:07.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.393+0000 7f52edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:07.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.393+0000 7f52edffb640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:05:07.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.393+0000 7f52edffb640 1 -- 172.21.15.71:0/1449146447 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52f00ae530 con 0x7f52f00b2800 2024-03-29T11:05:07.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.393+0000 7f52edffb640 1 -- 172.21.15.71:0/1449146447 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4264484558 0 0) 0x7f52dc004eb0 con 0x7f52f00a9c00 2024-03-29T11:05:07.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.394+0000 7f52edffb640 1 -- 172.21.15.71:0/1449146447 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52f01da670 con 0x7f52f00a9c00 2024-03-29T11:05:07.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.394+0000 7f52edffb640 1 -- 172.21.15.71:0/1449146447 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f52e8001d90 con 0x7f52f00a9c00 2024-03-29T11:05:07.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.394+0000 7f52edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:07.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.394+0000 7f52edffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:05:07.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.394+0000 7f52edffb640 10 monclient(hunting): dump: 2024-03-29T11:05:07.385 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:07.385 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:07.385 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:07.385 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:07.385 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:07.385 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:07.385 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:07.385 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:07.385 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:07.385 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:07.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.394+0000 7f52edffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:05:07.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.394+0000 7f52edffb640 1 -- 172.21.15.71:0/1449146447 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2671852990 0 0) 0x7f52f01d70f0 con 0x7f52f00b38c0 2024-03-29T11:05:07.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.394+0000 7f52edffb640 1 -- 172.21.15.71:0/1449146447 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52dc004eb0 con 0x7f52f00b38c0 2024-03-29T11:05:07.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.394+0000 7f52edffb640 1 -- 172.21.15.71:0/1449146447 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4230036043 0 0) 0x7f52f01da670 con 0x7f52f00a9c00 2024-03-29T11:05:07.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.394+0000 7f52edffb640 1 -- 172.21.15.71:0/1449146447 >> v1:172.21.15.137:6789/0 conn(0x7f52f00b2800 legacy=0x7f52f01dab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:07.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.394+0000 7f52edffb640 1 -- 172.21.15.71:0/1449146447 >> v1:172.21.15.71:6789/0 conn(0x7f52f00b38c0 legacy=0x7f52f01d7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:07.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.394+0000 7f52edffb640 10 monclient: _finish_hunting 0 2024-03-29T11:05:07.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.394+0000 7f52edffb640 1 monclient: found mon.c 2024-03-29T11:05:07.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.394+0000 7f52edffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:07.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.394+0000 7f52edffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:05:07.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.394+0000 7f52edffb640 1 -- 172.21.15.71:0/1449146447 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f00b0210 con 0x7f52f00a9c00 2024-03-29T11:05:07.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.394+0000 7f52edffb640 10 monclient: _finish_auth 0 2024-03-29T11:05:07.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.394+0000 7f52edffb640 10 monclient: _check_auth_tickets 2024-03-29T11:05:07.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.394+0000 7f52edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:07.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.394+0000 7f52edffb640 1 -- 172.21.15.71:0/1449146447 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52e8001ef0 con 0x7f52f00a9c00 2024-03-29T11:05:07.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.394+0000 7f52edffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:07.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.394+0000 7f52edffb640 1 -- 172.21.15.71:0/1449146447 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f52e8003de0 con 0x7f52f00a9c00 2024-03-29T11:05:07.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.394+0000 7f52ef7fe640 4 set_mon_vals no callback set 2024-03-29T11:05:07.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.394+0000 7f52f65f6640 5 monclient: authenticate success, global_id 4424 2024-03-29T11:05:07.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.395+0000 7f52f65f6640 10 monclient: _renew_subs 2024-03-29T11:05:07.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.395+0000 7f52f65f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:05:07.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.395+0000 7f52f65f6640 1 -- 172.21.15.71:0/1449146447 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f01db210 con 0x7f52f00a9c00 2024-03-29T11:05:07.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.395+0000 7f52f65f6640 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:05:07.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.395+0000 7f52f65f6640 10 monclient: _renew_subs 2024-03-29T11:05:07.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.395+0000 7f52f65f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:05:07.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.395+0000 7f52f65f6640 1 -- 172.21.15.71:0/1449146447 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f01d7c50 con 0x7f52f00a9c00 2024-03-29T11:05:07.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.395+0000 7f52f65f6640 1 librados: init done 2024-03-29T11:05:07.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.395+0000 7f52edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:07.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.395+0000 7f52edffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:05:07.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.395+0000 7f52edffb640 10 monclient: dump: 2024-03-29T11:05:07.388 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:07.388 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:07.388 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:07.388 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:07.388 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:07.388 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:07.388 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:07.388 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:07.388 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:07.389 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:07.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.397+0000 7f52d77fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:07.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.397+0000 7f52d77fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:07.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.397+0000 7f52d77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:05:07.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.397+0000 7f52d77fe640 1 -- 172.21.15.71:0/1449146447 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f004f730 con 0x7f52f00a9c00 2024-03-29T11:05:07.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.398+0000 7f52edffb640 1 -- 172.21.15.71:0/1449146447 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f52e8003f40 con 0x7f52f00a9c00 2024-03-29T11:05:07.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.398+0000 7f52edffb640 1 -- 172.21.15.71:0/1449146447 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f52e8084e50 con 0x7f52f00a9c00 2024-03-29T11:05:07.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.398+0000 7f52f4df3640 10 monclient: get_auth_request con 0x7f52dc06dd70 auth_method 0 2024-03-29T11:05:07.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.398+0000 7f52edffb640 10 client.4424.objecter ms_dispatch 0x7f52f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:05:07.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.399+0000 7f52edffb640 3 client.4424.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:05:07.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.399+0000 7f52edffb640 3 client.4424.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:05:07.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.399+0000 7f52edffb640 20 client.4424.objecter dump_active .. 0 homeless 2024-03-29T11:05:07.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.399+0000 7f52edffb640 10 client.4424.objecter ms_handle_connect 0x7f52dc06dd70 2024-03-29T11:05:07.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.407+0000 7f52edffb640 1 -- 172.21.15.71:0/1449146447 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f52e8050560 con 0x7f52f00a9c00 2024-03-29T11:05:07.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.407+0000 7f52edffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:07.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.407+0000 7f52edffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:05:07.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.774+0000 7f52d77fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:07.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.774+0000 7f52d77fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:07.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.774+0000 7f52d77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:05:07.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.774+0000 7f52d77fe640 1 -- 172.21.15.71:0/1449146447 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52f00b4980 con 0x7f52f00a9c00 2024-03-29T11:05:07.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.775+0000 7f52edffb640 1 -- 172.21.15.71:0/1449146447 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f52e8052660 con 0x7f52f00a9c00 2024-03-29T11:05:07.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.775+0000 7f52edffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:07.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.775+0000 7f52edffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:05:07.755 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:05:07.755 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:05:07.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.778+0000 7f52d77fe640 10 librados: watch_flush enter 2024-03-29T11:05:07.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.778+0000 7f52d77fe640 10 librados: watch_flush exit 2024-03-29T11:05:07.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.778+0000 7f52d77fe640 20 client.4424.objecter shutdown clearing up homeless session... 2024-03-29T11:05:07.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.778+0000 7f52d77fe640 10 client.4424.objecter successfully canceled tick 2024-03-29T11:05:07.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.779+0000 7f52d77fe640 1 -- 172.21.15.71:0/1449146447 >> v1:172.21.15.137:6816/973332665 conn(0x7f52dc06dd70 legacy=0x7f52dc070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:07.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.779+0000 7f52d77fe640 10 monclient: shutdown 2024-03-29T11:05:07.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.779+0000 7f52d77fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:07.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.779+0000 7f52d77fe640 1 -- 172.21.15.71:0/1449146447 >> v1:172.21.15.71:6790/0 conn(0x7f52f00a9c00 legacy=0x7f52f00ade20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:07.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.779+0000 7f52d77fe640 1 -- 172.21.15.71:0/1449146447 shutdown_connections 2024-03-29T11:05:07.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.779+0000 7f52d77fe640 1 -- 172.21.15.71:0/1449146447 >> 172.21.15.71:0/1449146447 conn(0x7f52f0066a00 msgr2=0x7f52f00b6ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:07.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.779+0000 7f52d77fe640 1 -- 172.21.15.71:0/1449146447 shutdown_connections 2024-03-29T11:05:07.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.779+0000 7f52d77fe640 1 -- 172.21.15.71:0/1449146447 wait complete. 2024-03-29T11:05:07.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:07.779+0000 7f52d77fe640 1 librados: shutdown 2024-03-29T11:05:07.773 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:05:14.580 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:05:14.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.749+0000 7fb5ad0e9640 10 monclient: get_monmap_and_config 2024-03-29T11:05:14.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.749+0000 7fb5ad0e9640 10 monclient: build_initial_monmap 2024-03-29T11:05:14.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.749+0000 7fb5ad0e9640 10 monclient: monmap: 2024-03-29T11:05:14.730 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:05:14.730 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:14.730 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:05:14.750771+0000 2024-03-29T11:05:14.730 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:05:14.750771+0000 2024-03-29T11:05:14.730 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:14.730 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:14.730 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:05:14.730 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:05:14.731 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:05:14.731 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:14.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.752+0000 7fb5ad0e9640 1 Processor -- start 2024-03-29T11:05:14.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.752+0000 7fb5ad0e9640 1 -- start start 2024-03-29T11:05:14.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.752+0000 7fb5ad0e9640 10 monclient: init 2024-03-29T11:05:14.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.753+0000 7fb5ad0e9640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:14.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.753+0000 7fb5ad0e9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:05:14.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.753+0000 7fb5ad0e9640 10 monclient(hunting): picked mon.noname-b con 0x7fb5a0006410 addr v1:172.21.15.71:6790/0 2024-03-29T11:05:14.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.753+0000 7fb5ad0e9640 10 monclient(hunting): picked mon.noname-c con 0x7fb5a002a640 addr v1:172.21.15.137:6789/0 2024-03-29T11:05:14.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.753+0000 7fb5ad0e9640 10 monclient(hunting): picked mon.noname-a con 0x7fb5a00327d0 addr v1:172.21.15.71:6789/0 2024-03-29T11:05:14.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.753+0000 7fb5ad0e9640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a002c820 con 0x7fb5a00327d0 2024-03-29T11:05:14.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.753+0000 7fb5ad0e9640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a002a120 con 0x7fb5a002a640 2024-03-29T11:05:14.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.753+0000 7fb5ad0e9640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a00c75b0 con 0x7fb5a0006410 2024-03-29T11:05:14.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.754+0000 7fb5ad0e9640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:14.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.754+0000 7fb5ad0e9640 10 monclient(hunting): authenticate will time out at 1071.012137s 2024-03-29T11:05:14.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.756+0000 7fb5a7fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fb5a0006410 0x7fb5a00c6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:54328/0 (socket says 172.21.15.71:54328) 2024-03-29T11:05:14.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.756+0000 7fb5a7fff640 1 -- 172.21.15.71:0/2075075292 learned_addr learned my addr 172.21.15.71:0/2075075292 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:05:14.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.756+0000 7fb5a6ffd640 1 -- 172.21.15.71:0/2075075292 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3681904972 0 0) 0x7fb5a00c75b0 con 0x7fb5a0006410 2024-03-29T11:05:14.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.756+0000 7fb5a6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:05:14.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.756+0000 7fb5a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:14.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.756+0000 7fb5a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:14.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.756+0000 7fb5a6ffd640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:05:14.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.756+0000 7fb5a6ffd640 1 -- 172.21.15.71:0/2075075292 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb598003520 con 0x7fb5a0006410 2024-03-29T11:05:14.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.757+0000 7fb5a6ffd640 1 -- 172.21.15.71:0/2075075292 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1048071181 0 0) 0x7fb598003520 con 0x7fb5a0006410 2024-03-29T11:05:14.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.757+0000 7fb5a6ffd640 1 -- 172.21.15.71:0/2075075292 >> v1:172.21.15.137:6789/0 conn(0x7fb5a002a640 legacy=0x7fb5a00320c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:05:14.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.757+0000 7fb5a6ffd640 1 -- 172.21.15.71:0/2075075292 >> v1:172.21.15.71:6789/0 conn(0x7fb5a00327d0 legacy=0x7fb5a00d0350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:05:14.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.758+0000 7fb5a6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:05:14.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.758+0000 7fb5a6ffd640 1 monclient: found mon.noname-b 2024-03-29T11:05:14.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.758+0000 7fb5a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:14.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.758+0000 7fb5a6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:05:14.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.758+0000 7fb5a6ffd640 1 -- 172.21.15.71:0/2075075292 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a0033110 con 0x7fb5a0006410 2024-03-29T11:05:14.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.758+0000 7fb5a6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:05:14.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.758+0000 7fb5a6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:05:14.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.758+0000 7fb5a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:14.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.758+0000 7fb5a6ffd640 1 -- 172.21.15.71:0/2075075292 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb59c001690 con 0x7fb5a0006410 2024-03-29T11:05:14.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.758+0000 7fb5a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:14.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.758+0000 7fb5a6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:05:14.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.758+0000 7fb5a6ffd640 10 monclient: dump: 2024-03-29T11:05:14.739 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:14.739 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:14.740 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:14.740 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:14.740 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:14.740 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:14.740 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:14.740 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:14.740 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:14.740 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:14.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.758+0000 7fb5ad0e9640 5 monclient: authenticate success, global_id 4427 2024-03-29T11:05:14.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.758+0000 7fb5ad0e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:05:14.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.758+0000 7fb5a6ffd640 1 -- 172.21.15.71:0/2075075292 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb59c0018f0 con 0x7fb5a0006410 2024-03-29T11:05:14.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.758+0000 7fb5a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:14.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.758+0000 7fb5a6ffd640 1 -- 172.21.15.71:0/2075075292 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb59c001d40 con 0x7fb5a0006410 2024-03-29T11:05:14.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.758+0000 7fb5a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:14.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.758+0000 7fb5a6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:05:14.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.758+0000 7fb5a6ffd640 10 monclient: dump: 2024-03-29T11:05:14.744 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:14.744 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:14.744 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:14.745 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:14.745 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:14.745 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:14.745 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:14.745 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:14.745 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:14.745 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:14.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.758+0000 7fb5ad0e9640 10 monclient: get_monmap_and_config success 2024-03-29T11:05:14.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.758+0000 7fb5ad0e9640 4 set_mon_vals no callback set 2024-03-29T11:05:14.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.759+0000 7fb5ad0e9640 10 monclient: shutdown 2024-03-29T11:05:14.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.759+0000 7fb5ad0e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:14.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.759+0000 7fb5ad0e9640 1 -- 172.21.15.71:0/2075075292 >> v1:172.21.15.71:6790/0 conn(0x7fb5a0006410 legacy=0x7fb5a00c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:14.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.760+0000 7fb5ad0e9640 1 -- 172.21.15.71:0/2075075292 shutdown_connections 2024-03-29T11:05:14.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.760+0000 7fb5ad0e9640 1 -- 172.21.15.71:0/2075075292 >> 172.21.15.71:0/2075075292 conn(0x7fb5a00058f0 msgr2=0x7fb5a0005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:14.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.760+0000 7fb5ad0e9640 1 -- 172.21.15.71:0/2075075292 shutdown_connections 2024-03-29T11:05:14.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.760+0000 7fb5ad0e9640 1 -- 172.21.15.71:0/2075075292 wait complete. 2024-03-29T11:05:14.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.761+0000 7fb5ad0e9640 10 monclient: build_initial_monmap 2024-03-29T11:05:14.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.761+0000 7fb5ad0e9640 10 monclient: monmap: 2024-03-29T11:05:14.746 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:05:14.746 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:05:14.746 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:05:14.746 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:05:14.746 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:14.746 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:14.747 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:05:14.747 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:05:14.747 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:05:14.747 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:14.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.761+0000 7fb5ad0e9640 1 librados: starting msgr at 2024-03-29T11:05:14.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.761+0000 7fb5ad0e9640 1 librados: starting objecter 2024-03-29T11:05:14.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.761+0000 7fb5ad0e9640 1 Processor -- start 2024-03-29T11:05:14.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.762+0000 7fb5ad0e9640 1 -- start start 2024-03-29T11:05:14.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.762+0000 7fb5ad0e9640 1 librados: setting wanted keys 2024-03-29T11:05:14.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.762+0000 7fb5ad0e9640 1 librados: calling monclient init 2024-03-29T11:05:14.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.762+0000 7fb5ad0e9640 10 monclient: init 2024-03-29T11:05:14.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.763+0000 7fb5ad0e9640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:14.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.763+0000 7fb5ad0e9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:05:14.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.763+0000 7fb5ad0e9640 10 monclient(hunting): picked mon.noname-c con 0x7fb5a0006410 addr v1:172.21.15.71:6790/0 2024-03-29T11:05:14.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.763+0000 7fb5ad0e9640 10 monclient(hunting): picked mon.noname-b con 0x7fb5a002a640 addr v1:172.21.15.137:6789/0 2024-03-29T11:05:14.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.763+0000 7fb5ad0e9640 10 monclient(hunting): picked mon.noname-a con 0x7fb5a00327d0 addr v1:172.21.15.71:6789/0 2024-03-29T11:05:14.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.763+0000 7fb5ad0e9640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a015c600 con 0x7fb5a00327d0 2024-03-29T11:05:14.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.763+0000 7fb5ad0e9640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a00c2780 con 0x7fb5a002a640 2024-03-29T11:05:14.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.763+0000 7fb5ad0e9640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a00c1720 con 0x7fb5a0006410 2024-03-29T11:05:14.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.763+0000 7fb5ad0e9640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:14.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.763+0000 7fb5ad0e9640 10 monclient(hunting): authenticate will time out at 1071.021510s 2024-03-29T11:05:14.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.764+0000 7fb5a7fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fb5a0006410 0x7fb5a0152690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:55810/0 (socket says 172.21.15.71:55810) 2024-03-29T11:05:14.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.764+0000 7fb5a7fff640 1 -- 172.21.15.71:0/2794682391 learned_addr learned my addr 172.21.15.71:0/2794682391 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:05:14.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.766+0000 7fb5a4ff9640 10 client.?.objecter ms_handle_connect 0x7fb5a0006410 2024-03-29T11:05:14.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.766+0000 7fb5a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:14.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.766+0000 7fb5a4ff9640 10 client.?.objecter ms_handle_connect 0x7fb5a00327d0 2024-03-29T11:05:14.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.766+0000 7fb5a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:14.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.766+0000 7fb5a4ff9640 1 -- 172.21.15.71:0/2794682391 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2882505110 0 0) 0x7fb5a00c1720 con 0x7fb5a0006410 2024-03-29T11:05:14.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.766+0000 7fb5a4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:05:14.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.766+0000 7fb5a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:14.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.766+0000 7fb5a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:14.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.766+0000 7fb5a4ff9640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:05:14.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.767+0000 7fb5a4ff9640 1 -- 172.21.15.71:0/2794682391 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb58c004eb0 con 0x7fb5a0006410 2024-03-29T11:05:14.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.767+0000 7fb5a4ff9640 1 -- 172.21.15.71:0/2794682391 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 701254462 0 0) 0x7fb5a015c600 con 0x7fb5a00327d0 2024-03-29T11:05:14.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.767+0000 7fb5a4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:05:14.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.767+0000 7fb5a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:14.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.767+0000 7fb5a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:14.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.767+0000 7fb5a4ff9640 10 monclient(hunting): my global_id is 4413 2024-03-29T11:05:14.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.767+0000 7fb5a4ff9640 1 -- 172.21.15.71:0/2794682391 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5a00c1720 con 0x7fb5a00327d0 2024-03-29T11:05:14.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.768+0000 7fb5a4ff9640 1 -- 172.21.15.71:0/2794682391 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2500983699 0 0) 0x7fb58c004eb0 con 0x7fb5a0006410 2024-03-29T11:05:14.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.768+0000 7fb5a4ff9640 1 -- 172.21.15.71:0/2794682391 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5a015c600 con 0x7fb5a0006410 2024-03-29T11:05:14.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.768+0000 7fb5a4ff9640 1 -- 172.21.15.71:0/2794682391 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3486551825 0 0) 0x7fb5a00c1720 con 0x7fb5a00327d0 2024-03-29T11:05:14.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.768+0000 7fb5a4ff9640 1 -- 172.21.15.71:0/2794682391 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb58c004eb0 con 0x7fb5a00327d0 2024-03-29T11:05:14.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.768+0000 7fb5a4ff9640 1 -- 172.21.15.71:0/2794682391 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb59c0014b0 con 0x7fb5a0006410 2024-03-29T11:05:14.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.769+0000 7fb5a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:14.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.769+0000 7fb5a4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:05:14.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.771+0000 7fb5a4ff9640 10 monclient(hunting): dump: 2024-03-29T11:05:14.753 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:14.753 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:14.753 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:14.753 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:14.753 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:14.753 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:14.753 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:14.753 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:14.754 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:14.754 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:14.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.771+0000 7fb5a4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:05:14.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.771+0000 7fb5a4ff9640 10 client.?.objecter ms_handle_connect 0x7fb5a002a640 2024-03-29T11:05:14.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.771+0000 7fb5a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:14.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.771+0000 7fb5a4ff9640 1 -- 172.21.15.71:0/2794682391 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb5a804fdd0 con 0x7fb5a00327d0 2024-03-29T11:05:14.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.771+0000 7fb5a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:14.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.771+0000 7fb5a4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:05:14.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.771+0000 7fb5a4ff9640 10 monclient(hunting): dump: 2024-03-29T11:05:14.754 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:14.754 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:14.754 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:14.754 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:14.754 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:14.755 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:14.755 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:14.755 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:14.755 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:14.755 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:14.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.771+0000 7fb5a4ff9640 1 -- 172.21.15.71:0/2794682391 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1522310633 0 0) 0x7fb5a015c600 con 0x7fb5a0006410 2024-03-29T11:05:14.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.771+0000 7fb5a4ff9640 1 -- 172.21.15.71:0/2794682391 >> v1:172.21.15.137:6789/0 conn(0x7fb5a002a640 legacy=0x7fb5a015ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:14.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.772+0000 7fb5a4ff9640 1 -- 172.21.15.71:0/2794682391 >> v1:172.21.15.71:6789/0 conn(0x7fb5a00327d0 legacy=0x7fb5a0166f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:14.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.772+0000 7fb5a4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:05:14.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.772+0000 7fb5a4ff9640 1 monclient: found mon.c 2024-03-29T11:05:14.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.772+0000 7fb5a4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:14.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.772+0000 7fb5a4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:05:14.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.772+0000 7fb5a4ff9640 1 -- 172.21.15.71:0/2794682391 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a0152e60 con 0x7fb5a0006410 2024-03-29T11:05:14.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.772+0000 7fb5a4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:05:14.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.772+0000 7fb5a4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:05:14.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.772+0000 7fb5a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:14.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.772+0000 7fb5ad0e9640 5 monclient: authenticate success, global_id 4430 2024-03-29T11:05:14.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.772+0000 7fb5ad0e9640 10 monclient: _renew_subs 2024-03-29T11:05:14.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.772+0000 7fb5ad0e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:05:14.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.772+0000 7fb5ad0e9640 1 -- 172.21.15.71:0/2794682391 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a0151bb0 con 0x7fb5a0006410 2024-03-29T11:05:14.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.772+0000 7fb5ad0e9640 10 client.4430.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:05:14.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.772+0000 7fb5ad0e9640 10 monclient: _renew_subs 2024-03-29T11:05:14.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.772+0000 7fb5ad0e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:05:14.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.772+0000 7fb5ad0e9640 1 -- 172.21.15.71:0/2794682391 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a015d1a0 con 0x7fb5a0006410 2024-03-29T11:05:14.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.772+0000 7fb5ad0e9640 1 librados: init done 2024-03-29T11:05:14.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.772+0000 7fb5a4ff9640 1 -- 172.21.15.71:0/2794682391 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb59c001610 con 0x7fb5a0006410 2024-03-29T11:05:14.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.772+0000 7fb5a4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:14.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.772+0000 7fb5a4ff9640 1 -- 172.21.15.71:0/2794682391 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb59c004e10 con 0x7fb5a0006410 2024-03-29T11:05:14.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.772+0000 7fb5a4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:14.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.773+0000 7fb5a4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:05:14.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.773+0000 7fb5a4ff9640 10 monclient: dump: 2024-03-29T11:05:14.758 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:14.758 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:14.759 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:14.759 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:14.759 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:14.759 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:14.759 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:14.759 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:14.759 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:14.759 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:14.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.773+0000 7fb5a67fc640 4 set_mon_vals no callback set 2024-03-29T11:05:14.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.774+0000 7fb5ad0e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:14.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.775+0000 7fb5ad0e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:14.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.775+0000 7fb5a4ff9640 1 -- 172.21.15.71:0/2794682391 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7fb59c004f70 con 0x7fb5a0006410 2024-03-29T11:05:14.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.775+0000 7fb5ad0e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:05:14.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.775+0000 7fb5ad0e9640 1 -- 172.21.15.71:0/2794682391 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb588005080 con 0x7fb5a0006410 2024-03-29T11:05:14.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.776+0000 7fb5a4ff9640 1 -- 172.21.15.71:0/2794682391 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7fb59c084cf0 con 0x7fb5a0006410 2024-03-29T11:05:14.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.776+0000 7fb5a4ff9640 10 client.4430.objecter ms_dispatch 0x7fb5a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:05:14.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.776+0000 7fb5a4ff9640 3 client.4430.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:05:14.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.776+0000 7fb5a4ff9640 3 client.4430.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:05:14.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.776+0000 7fb5a4ff9640 20 client.4430.objecter dump_active .. 0 homeless 2024-03-29T11:05:14.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.777+0000 7fb5a77fe640 10 monclient: get_auth_request con 0x7fb58c0766f0 auth_method 0 2024-03-29T11:05:14.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.777+0000 7fb5a4ff9640 10 client.4430.objecter ms_handle_connect 0x7fb58c0766f0 2024-03-29T11:05:14.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.786+0000 7fb5a4ff9640 1 -- 172.21.15.71:0/2794682391 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb59c0505d0 con 0x7fb5a0006410 2024-03-29T11:05:14.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.786+0000 7fb5a4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:14.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:14.786+0000 7fb5a4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:05:15.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:15.108+0000 7fb5ad0e9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:15.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:15.108+0000 7fb5ad0e9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:15.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:15.108+0000 7fb5ad0e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:05:15.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:15.108+0000 7fb5ad0e9640 1 -- 172.21.15.71:0/2794682391 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb588005330 con 0x7fb5a0006410 2024-03-29T11:05:15.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:15.109+0000 7fb5a4ff9640 1 -- 172.21.15.71:0/2794682391 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7fb59c052500 con 0x7fb5a0006410 2024-03-29T11:05:15.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:15.109+0000 7fb5a4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:15.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:15.109+0000 7fb5a4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:05:15.089 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:05:15.089 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:05:15.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:15.112+0000 7fb5ad0e9640 10 librados: watch_flush enter 2024-03-29T11:05:15.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:15.112+0000 7fb5ad0e9640 10 librados: watch_flush exit 2024-03-29T11:05:15.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:15.112+0000 7fb5ad0e9640 20 client.4430.objecter shutdown clearing up homeless session... 2024-03-29T11:05:15.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:15.112+0000 7fb5ad0e9640 10 client.4430.objecter successfully canceled tick 2024-03-29T11:05:15.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:15.112+0000 7fb5ad0e9640 1 -- 172.21.15.71:0/2794682391 >> v1:172.21.15.137:6816/973332665 conn(0x7fb58c0766f0 legacy=0x7fb58c078bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:15.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:15.113+0000 7fb5ad0e9640 10 monclient: shutdown 2024-03-29T11:05:15.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:15.113+0000 7fb5ad0e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:15.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:15.113+0000 7fb5ad0e9640 1 -- 172.21.15.71:0/2794682391 >> v1:172.21.15.71:6790/0 conn(0x7fb5a0006410 legacy=0x7fb5a0152690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:15.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:15.113+0000 7fb5ad0e9640 1 -- 172.21.15.71:0/2794682391 shutdown_connections 2024-03-29T11:05:15.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:15.113+0000 7fb5ad0e9640 1 -- 172.21.15.71:0/2794682391 >> 172.21.15.71:0/2794682391 conn(0x7fb5a00058f0 msgr2=0x7fb5a00043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:15.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:15.113+0000 7fb5ad0e9640 1 -- 172.21.15.71:0/2794682391 shutdown_connections 2024-03-29T11:05:15.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:15.113+0000 7fb5ad0e9640 1 -- 172.21.15.71:0/2794682391 wait complete. 2024-03-29T11:05:15.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:15.113+0000 7fb5ad0e9640 1 librados: shutdown 2024-03-29T11:05:15.106 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:05:21.914 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:05:22.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.088+0000 7effddbc8640 10 monclient: get_monmap_and_config 2024-03-29T11:05:22.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.088+0000 7effddbc8640 10 monclient: build_initial_monmap 2024-03-29T11:05:22.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.088+0000 7effddbc8640 10 monclient: monmap: 2024-03-29T11:05:22.068 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:05:22.068 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:22.068 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:05:22.089436+0000 2024-03-29T11:05:22.068 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:05:22.089436+0000 2024-03-29T11:05:22.068 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:22.068 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:22.068 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:05:22.068 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:05:22.068 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:05:22.068 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:22.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.091+0000 7effddbc8640 1 Processor -- start 2024-03-29T11:05:22.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.091+0000 7effddbc8640 1 -- start start 2024-03-29T11:05:22.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.091+0000 7effddbc8640 10 monclient: init 2024-03-29T11:05:22.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.092+0000 7effddbc8640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:22.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.092+0000 7effddbc8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:05:22.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.092+0000 7effddbc8640 10 monclient(hunting): picked mon.noname-c con 0x7effd80b1f60 addr v1:172.21.15.137:6789/0 2024-03-29T11:05:22.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.092+0000 7effddbc8640 10 monclient(hunting): picked mon.noname-b con 0x7effd8071eb0 addr v1:172.21.15.71:6790/0 2024-03-29T11:05:22.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.092+0000 7effddbc8640 10 monclient(hunting): picked mon.noname-a con 0x7effd8143d10 addr v1:172.21.15.71:6789/0 2024-03-29T11:05:22.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.092+0000 7effddbc8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd806ef80 con 0x7effd8143d10 2024-03-29T11:05:22.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.092+0000 7effddbc8640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd806f260 con 0x7effd80b1f60 2024-03-29T11:05:22.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.092+0000 7effddbc8640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd80b40c0 con 0x7effd8071eb0 2024-03-29T11:05:22.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.092+0000 7effddbc8640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:22.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.092+0000 7effddbc8640 10 monclient(hunting): authenticate will time out at 1078.350764s 2024-03-29T11:05:22.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.097+0000 7effd7fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7effd8143d10 0x7effd8146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:54409/0 (socket says 172.21.15.71:54409) 2024-03-29T11:05:22.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.097+0000 7effd7fff640 1 -- 172.21.15.71:0/3000441688 learned_addr learned my addr 172.21.15.71:0/3000441688 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:05:22.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.100+0000 7effd67fc640 1 -- 172.21.15.71:0/3000441688 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685666526 0 0) 0x7effd806f260 con 0x7effd80b1f60 2024-03-29T11:05:22.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.100+0000 7effd67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:05:22.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.100+0000 7effd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:22.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.100+0000 7effd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:22.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.100+0000 7effd67fc640 10 monclient(hunting): my global_id is 4381 2024-03-29T11:05:22.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.100+0000 7effd67fc640 1 -- 172.21.15.71:0/3000441688 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effc0003520 con 0x7effd80b1f60 2024-03-29T11:05:22.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 1 -- 172.21.15.71:0/3000441688 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154361497 0 0) 0x7effd80b40c0 con 0x7effd8071eb0 2024-03-29T11:05:22.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:05:22.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:22.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:22.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:05:22.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 1 -- 172.21.15.71:0/3000441688 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effd806f260 con 0x7effd8071eb0 2024-03-29T11:05:22.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 1 -- 172.21.15.71:0/3000441688 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2877534525 0 0) 0x7effd806ef80 con 0x7effd8143d10 2024-03-29T11:05:22.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:05:22.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:22.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:22.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 10 monclient(hunting): my global_id is 4416 2024-03-29T11:05:22.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 1 -- 172.21.15.71:0/3000441688 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effd80b40c0 con 0x7effd8143d10 2024-03-29T11:05:22.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 1 -- 172.21.15.71:0/3000441688 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 773654930 0 0) 0x7effd80b40c0 con 0x7effd8143d10 2024-03-29T11:05:22.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 1 -- 172.21.15.71:0/3000441688 >> v1:172.21.15.71:6790/0 conn(0x7effd8071eb0 legacy=0x7effd8146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:22.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 1 -- 172.21.15.71:0/3000441688 >> v1:172.21.15.137:6789/0 conn(0x7effd80b1f60 legacy=0x7effd8143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:22.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:05:22.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 1 monclient: found mon.noname-a 2024-03-29T11:05:22.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:22.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:05:22.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 1 -- 172.21.15.71:0/3000441688 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd8155d70 con 0x7effd8143d10 2024-03-29T11:05:22.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 10 monclient: _finish_auth 0 2024-03-29T11:05:22.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:05:22.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:22.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effddbc8640 5 monclient: authenticate success, global_id 4416 2024-03-29T11:05:22.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effddbc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:05:22.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 1 -- 172.21.15.71:0/3000441688 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7effcc001530 con 0x7effd8143d10 2024-03-29T11:05:22.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:22.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:05:22.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 10 monclient: dump: 2024-03-29T11:05:22.083 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:22.083 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:22.083 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:22.083 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:22.083 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:22.083 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:22.083 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:22.083 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:22.084 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:22.084 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:22.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effddbc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:05:22.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 1 -- 172.21.15.71:0/3000441688 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7effcc001750 con 0x7effd8143d10 2024-03-29T11:05:22.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effd67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:22.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effddbc8640 10 monclient: get_monmap_and_config success 2024-03-29T11:05:22.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.101+0000 7effddbc8640 4 set_mon_vals no callback set 2024-03-29T11:05:22.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.102+0000 7effd67fc640 1 -- 172.21.15.71:0/3000441688 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7effcc001ae0 con 0x7effd8143d10 2024-03-29T11:05:22.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.102+0000 7effd67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:22.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.102+0000 7effd67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:05:22.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.102+0000 7effd67fc640 10 monclient: dump: 2024-03-29T11:05:22.085 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:22.085 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:22.085 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:22.085 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:22.085 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:22.085 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:22.085 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:22.085 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:22.086 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:22.086 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:22.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.102+0000 7effddbc8640 10 monclient: shutdown 2024-03-29T11:05:22.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.102+0000 7effddbc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:22.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.102+0000 7effddbc8640 1 -- 172.21.15.71:0/3000441688 >> v1:172.21.15.71:6789/0 conn(0x7effd8143d10 legacy=0x7effd8146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:22.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.103+0000 7effddbc8640 1 -- 172.21.15.71:0/3000441688 shutdown_connections 2024-03-29T11:05:22.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.103+0000 7effddbc8640 1 -- 172.21.15.71:0/3000441688 >> 172.21.15.71:0/3000441688 conn(0x7effd8066a00 msgr2=0x7effd8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:22.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.105+0000 7effddbc8640 1 -- 172.21.15.71:0/3000441688 shutdown_connections 2024-03-29T11:05:22.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.105+0000 7effddbc8640 1 -- 172.21.15.71:0/3000441688 wait complete. 2024-03-29T11:05:22.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.106+0000 7effddbc8640 10 monclient: build_initial_monmap 2024-03-29T11:05:22.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.106+0000 7effddbc8640 10 monclient: monmap: 2024-03-29T11:05:22.087 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:05:22.087 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:05:22.088 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:05:22.088 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:05:22.088 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:22.088 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:22.088 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:05:22.088 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:05:22.088 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:05:22.088 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:22.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.107+0000 7effddbc8640 1 librados: starting msgr at 2024-03-29T11:05:22.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.107+0000 7effddbc8640 1 librados: starting objecter 2024-03-29T11:05:22.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.107+0000 7effddbc8640 1 Processor -- start 2024-03-29T11:05:22.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.107+0000 7effddbc8640 1 -- start start 2024-03-29T11:05:22.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.107+0000 7effddbc8640 1 librados: setting wanted keys 2024-03-29T11:05:22.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.107+0000 7effddbc8640 1 librados: calling monclient init 2024-03-29T11:05:22.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.107+0000 7effddbc8640 10 monclient: init 2024-03-29T11:05:22.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.108+0000 7effddbc8640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:22.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.108+0000 7effddbc8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:05:22.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.110+0000 7effddbc8640 10 monclient(hunting): picked mon.noname-b con 0x7effd8143d10 addr v1:172.21.15.137:6789/0 2024-03-29T11:05:22.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.110+0000 7effddbc8640 10 monclient(hunting): picked mon.noname-c con 0x7effd808b310 addr v1:172.21.15.71:6790/0 2024-03-29T11:05:22.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.110+0000 7effddbc8640 10 monclient(hunting): picked mon.noname-a con 0x7effd8149ad0 addr v1:172.21.15.71:6789/0 2024-03-29T11:05:22.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.110+0000 7effddbc8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd808b900 con 0x7effd8149ad0 2024-03-29T11:05:22.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.110+0000 7effddbc8640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd808bdb0 con 0x7effd8143d10 2024-03-29T11:05:22.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.110+0000 7effddbc8640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd814b6b0 con 0x7effd808b310 2024-03-29T11:05:22.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.110+0000 7effddbc8640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:22.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.110+0000 7effddbc8640 10 monclient(hunting): authenticate will time out at 1078.368303s 2024-03-29T11:05:22.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.116+0000 7effd6ffd640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7effd808b310 0x7effd81493c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:42775/0 (socket says 172.21.15.71:42775) 2024-03-29T11:05:22.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.116+0000 7effd6ffd640 1 -- 172.21.15.71:0/2360969533 learned_addr learned my addr 172.21.15.71:0/2360969533 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:05:22.096 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.116+0000 7effbffff640 10 client.?.objecter ms_handle_connect 0x7effd808b310 2024-03-29T11:05:22.096 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.116+0000 7effbffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:22.097 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.117+0000 7effbffff640 1 -- 172.21.15.71:0/2360969533 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3367996341 0 0) 0x7effd814b6b0 con 0x7effd808b310 2024-03-29T11:05:22.097 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.117+0000 7effbffff640 10 monclient(hunting): _negotiate 2024-03-29T11:05:22.097 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.117+0000 7effbffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:22.097 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.117+0000 7effbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:22.097 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.117+0000 7effbffff640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:05:22.097 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.117+0000 7effbffff640 1 -- 172.21.15.71:0/2360969533 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effc4004eb0 con 0x7effd808b310 2024-03-29T11:05:22.097 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.117+0000 7effbffff640 10 client.?.objecter ms_handle_connect 0x7effd8143d10 2024-03-29T11:05:22.097 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.117+0000 7effbffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:22.097 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.117+0000 7effbffff640 1 -- 172.21.15.71:0/2360969533 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 963583689 0 0) 0x7effd808bdb0 con 0x7effd8143d10 2024-03-29T11:05:22.098 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.117+0000 7effbffff640 10 monclient(hunting): _negotiate 2024-03-29T11:05:22.098 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.117+0000 7effbffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:22.098 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.117+0000 7effbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:22.098 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.117+0000 7effbffff640 10 monclient(hunting): my global_id is 4384 2024-03-29T11:05:22.098 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.117+0000 7effbffff640 1 -- 172.21.15.71:0/2360969533 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effd814b6b0 con 0x7effd8143d10 2024-03-29T11:05:22.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.121+0000 7effbffff640 1 -- 172.21.15.71:0/2360969533 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2801019901 0 0) 0x7effd814b6b0 con 0x7effd8143d10 2024-03-29T11:05:22.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.121+0000 7effbffff640 1 -- 172.21.15.71:0/2360969533 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7effd808bdb0 con 0x7effd8143d10 2024-03-29T11:05:22.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.121+0000 7effbffff640 1 -- 172.21.15.71:0/2360969533 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7effd0001d30 con 0x7effd8143d10 2024-03-29T11:05:22.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.121+0000 7effbffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:22.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.121+0000 7effbffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:05:22.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.121+0000 7effbffff640 10 monclient(hunting): dump: 2024-03-29T11:05:22.100 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:22.101 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:22.101 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:22.101 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:22.101 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:22.101 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:22.101 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:22.101 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:22.101 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:22.101 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:22.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.121+0000 7effbffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:05:22.104 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.122+0000 7effbffff640 1 -- 172.21.15.71:0/2360969533 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2999810005 0 0) 0x7effc4004eb0 con 0x7effd808b310 2024-03-29T11:05:22.104 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.122+0000 7effbffff640 1 -- 172.21.15.71:0/2360969533 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7effd814b6b0 con 0x7effd808b310 2024-03-29T11:05:22.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.122+0000 7effbffff640 1 -- 172.21.15.71:0/2360969533 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2345587855 0 0) 0x7effd808bdb0 con 0x7effd8143d10 2024-03-29T11:05:22.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.122+0000 7effbffff640 1 -- 172.21.15.71:0/2360969533 >> v1:172.21.15.71:6790/0 conn(0x7effd808b310 legacy=0x7effd81493c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:22.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.122+0000 7effbffff640 1 -- 172.21.15.71:0/2360969533 >> v1:172.21.15.71:6789/0 conn(0x7effd8149ad0 legacy=0x7effd813b990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:05:22.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.123+0000 7effbffff640 10 monclient: _finish_hunting 0 2024-03-29T11:05:22.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.123+0000 7effbffff640 1 monclient: found mon.b 2024-03-29T11:05:22.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.123+0000 7effbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:22.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.123+0000 7effbffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:05:22.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.123+0000 7effbffff640 1 -- 172.21.15.71:0/2360969533 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd814b990 con 0x7effd8143d10 2024-03-29T11:05:22.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.123+0000 7effbffff640 10 monclient: _finish_auth 0 2024-03-29T11:05:22.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.123+0000 7effbffff640 10 monclient: _check_auth_tickets 2024-03-29T11:05:22.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.123+0000 7effbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:22.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.123+0000 7effddbc8640 5 monclient: authenticate success, global_id 4384 2024-03-29T11:05:22.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.123+0000 7effddbc8640 10 monclient: _renew_subs 2024-03-29T11:05:22.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.123+0000 7effddbc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:05:22.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.123+0000 7effddbc8640 1 -- 172.21.15.71:0/2360969533 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effd8149f60 con 0x7effd8143d10 2024-03-29T11:05:22.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.123+0000 7effddbc8640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:05:22.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.123+0000 7effddbc8640 10 monclient: _renew_subs 2024-03-29T11:05:22.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.123+0000 7effddbc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:05:22.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.123+0000 7effddbc8640 1 -- 172.21.15.71:0/2360969533 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7effd813d290 con 0x7effd8143d10 2024-03-29T11:05:22.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.123+0000 7effddbc8640 1 librados: init done 2024-03-29T11:05:22.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.123+0000 7effbffff640 1 -- 172.21.15.71:0/2360969533 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7effd0001ff0 con 0x7effd8143d10 2024-03-29T11:05:22.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.123+0000 7effbffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:22.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.123+0000 7effbffff640 1 -- 172.21.15.71:0/2360969533 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7effd0003de0 con 0x7effd8143d10 2024-03-29T11:05:22.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.123+0000 7effbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:22.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.123+0000 7effbffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:05:22.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.123+0000 7effbffff640 10 monclient: dump: 2024-03-29T11:05:22.107 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:22.107 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:22.107 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:22.107 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:22.107 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:22.107 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:22.107 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:22.107 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:22.107 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:22.107 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:22.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.123+0000 7effd57fa640 4 set_mon_vals no callback set 2024-03-29T11:05:22.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.124+0000 7effddbc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:22.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.125+0000 7effddbc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:22.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.125+0000 7effddbc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:05:22.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.125+0000 7effddbc8640 1 -- 172.21.15.71:0/2360969533 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effd8075f10 con 0x7effd8143d10 2024-03-29T11:05:22.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.126+0000 7effbffff640 1 -- 172.21.15.71:0/2360969533 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7effd0019db0 con 0x7effd8143d10 2024-03-29T11:05:22.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.127+0000 7effbffff640 1 -- 172.21.15.71:0/2360969533 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7effd0083ac0 con 0x7effd8143d10 2024-03-29T11:05:22.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.127+0000 7effbffff640 10 client.4384.objecter ms_dispatch 0x7effd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:05:22.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.127+0000 7effbffff640 3 client.4384.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:05:22.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.127+0000 7effbffff640 3 client.4384.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:05:22.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.127+0000 7effbffff640 20 client.4384.objecter dump_active .. 0 homeless 2024-03-29T11:05:22.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.128+0000 7effd6ffd640 10 monclient: get_auth_request con 0x7effc406dd70 auth_method 0 2024-03-29T11:05:22.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.130+0000 7effbffff640 10 client.4384.objecter ms_handle_connect 0x7effc406dd70 2024-03-29T11:05:22.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.135+0000 7effbffff640 1 -- 172.21.15.71:0/2360969533 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7effd004f3a0 con 0x7effd8143d10 2024-03-29T11:05:22.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.135+0000 7effbffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:22.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.135+0000 7effbffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:05:22.433 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.453+0000 7effddbc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:22.433 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.453+0000 7effddbc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:22.433 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.453+0000 7effddbc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:05:22.433 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.453+0000 7effddbc8640 1 -- 172.21.15.71:0/2360969533 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7effd808b770 con 0x7effd8143d10 2024-03-29T11:05:22.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.457+0000 7effbffff640 1 -- 172.21.15.71:0/2360969533 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7effd00512d0 con 0x7effd8143d10 2024-03-29T11:05:22.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.457+0000 7effbffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:22.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.457+0000 7effbffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:05:22.437 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:05:22.437 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:05:22.441 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.461+0000 7effddbc8640 10 librados: watch_flush enter 2024-03-29T11:05:22.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.461+0000 7effddbc8640 10 librados: watch_flush exit 2024-03-29T11:05:22.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.461+0000 7effddbc8640 20 client.4384.objecter shutdown clearing up homeless session... 2024-03-29T11:05:22.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.462+0000 7effddbc8640 10 client.4384.objecter successfully canceled tick 2024-03-29T11:05:22.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.462+0000 7effddbc8640 1 -- 172.21.15.71:0/2360969533 >> v1:172.21.15.137:6816/973332665 conn(0x7effc406dd70 legacy=0x7effc4070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:22.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.462+0000 7effddbc8640 10 monclient: shutdown 2024-03-29T11:05:22.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.462+0000 7effddbc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:22.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.462+0000 7effddbc8640 1 -- 172.21.15.71:0/2360969533 >> v1:172.21.15.137:6789/0 conn(0x7effd8143d10 legacy=0x7effd8154dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:22.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.462+0000 7effddbc8640 1 -- 172.21.15.71:0/2360969533 shutdown_connections 2024-03-29T11:05:22.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.462+0000 7effddbc8640 1 -- 172.21.15.71:0/2360969533 >> 172.21.15.71:0/2360969533 conn(0x7effd8066a00 msgr2=0x7effd8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:22.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.462+0000 7effddbc8640 1 -- 172.21.15.71:0/2360969533 shutdown_connections 2024-03-29T11:05:22.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.462+0000 7effddbc8640 1 -- 172.21.15.71:0/2360969533 wait complete. 2024-03-29T11:05:22.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:22.463+0000 7effddbc8640 1 librados: shutdown 2024-03-29T11:05:22.459 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:05:29.266 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:05:29.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.431+0000 7faa45b4d640 10 monclient: get_monmap_and_config 2024-03-29T11:05:29.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.432+0000 7faa45b4d640 10 monclient: build_initial_monmap 2024-03-29T11:05:29.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.432+0000 7faa45b4d640 10 monclient: monmap: 2024-03-29T11:05:29.412 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:05:29.412 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:29.413 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:05:29.432872+0000 2024-03-29T11:05:29.413 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:05:29.432872+0000 2024-03-29T11:05:29.413 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:29.413 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:29.413 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:05:29.413 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:05:29.413 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:05:29.413 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:29.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.433+0000 7faa45b4d640 1 Processor -- start 2024-03-29T11:05:29.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.433+0000 7faa45b4d640 1 -- start start 2024-03-29T11:05:29.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.433+0000 7faa45b4d640 10 monclient: init 2024-03-29T11:05:29.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.434+0000 7faa45b4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:29.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.434+0000 7faa45b4d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:05:29.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.434+0000 7faa45b4d640 10 monclient(hunting): picked mon.noname-b con 0x7faa400a9ac0 addr v1:172.21.15.71:6790/0 2024-03-29T11:05:29.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.434+0000 7faa45b4d640 10 monclient(hunting): picked mon.noname-a con 0x7faa4013ee90 addr v1:172.21.15.71:6789/0 2024-03-29T11:05:29.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.434+0000 7faa45b4d640 10 monclient(hunting): picked mon.noname-c con 0x7faa400af900 addr v1:172.21.15.137:6789/0 2024-03-29T11:05:29.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.434+0000 7faa45b4d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa400afd10 con 0x7faa4013ee90 2024-03-29T11:05:29.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.434+0000 7faa45b4d640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa400af6f0 con 0x7faa400af900 2024-03-29T11:05:29.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.434+0000 7faa45b4d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa400aac80 con 0x7faa400a9ac0 2024-03-29T11:05:29.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.434+0000 7faa45b4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:29.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.434+0000 7faa45b4d640 10 monclient(hunting): authenticate will time out at 1085.692412s 2024-03-29T11:05:29.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.436+0000 7faa3f7fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7faa400a9ac0 0x7faa400adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:46764/0 (socket says 172.21.15.71:46764) 2024-03-29T11:05:29.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.436+0000 7faa3f7fe640 1 -- 172.21.15.71:0/765265544 learned_addr learned my addr 172.21.15.71:0/765265544 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:05:29.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.437+0000 7faa3e7fc640 1 -- 172.21.15.71:0/765265544 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3292161796 0 0) 0x7faa400aac80 con 0x7faa400a9ac0 2024-03-29T11:05:29.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.437+0000 7faa3e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:05:29.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.437+0000 7faa3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:29.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.437+0000 7faa3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:29.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.437+0000 7faa3e7fc640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:05:29.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.437+0000 7faa3e7fc640 1 -- 172.21.15.71:0/765265544 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa28003520 con 0x7faa400a9ac0 2024-03-29T11:05:29.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.437+0000 7faa3e7fc640 1 -- 172.21.15.71:0/765265544 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2977413772 0 0) 0x7faa28003520 con 0x7faa400a9ac0 2024-03-29T11:05:29.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.437+0000 7faa3e7fc640 1 -- 172.21.15.71:0/765265544 >> v1:172.21.15.137:6789/0 conn(0x7faa400af900 legacy=0x7faa40149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:29.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.437+0000 7faa3e7fc640 1 -- 172.21.15.71:0/765265544 >> v1:172.21.15.71:6789/0 conn(0x7faa4013ee90 legacy=0x7faa4013b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:05:29.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.438+0000 7faa3e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:05:29.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.438+0000 7faa3e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:05:29.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.438+0000 7faa3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:29.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.438+0000 7faa3e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:05:29.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.438+0000 7faa3e7fc640 1 -- 172.21.15.71:0/765265544 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa400aaf60 con 0x7faa400a9ac0 2024-03-29T11:05:29.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.438+0000 7faa3e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:05:29.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.438+0000 7faa3e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:05:29.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.438+0000 7faa3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:29.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.438+0000 7faa45b4d640 5 monclient: authenticate success, global_id 4439 2024-03-29T11:05:29.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.438+0000 7faa3e7fc640 1 -- 172.21.15.71:0/765265544 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7faa30001690 con 0x7faa400a9ac0 2024-03-29T11:05:29.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.438+0000 7faa3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:29.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.438+0000 7faa3e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:05:29.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.438+0000 7faa3e7fc640 10 monclient: dump: 2024-03-29T11:05:29.422 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:29.422 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:29.422 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:29.422 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:29.422 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:29.422 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:29.422 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:29.422 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:29.422 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:29.422 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:29.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.438+0000 7faa45b4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:05:29.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.439+0000 7faa3e7fc640 1 -- 172.21.15.71:0/765265544 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa30001920 con 0x7faa400a9ac0 2024-03-29T11:05:29.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.439+0000 7faa3e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:29.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.439+0000 7faa3e7fc640 1 -- 172.21.15.71:0/765265544 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7faa30001cb0 con 0x7faa400a9ac0 2024-03-29T11:05:29.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.439+0000 7faa3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:29.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.439+0000 7faa3e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:05:29.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.439+0000 7faa3e7fc640 10 monclient: dump: 2024-03-29T11:05:29.423 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:29.423 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:29.423 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:29.423 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:29.423 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:29.423 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:29.423 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:29.423 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:29.423 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:29.423 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:29.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.439+0000 7faa45b4d640 10 monclient: get_monmap_and_config success 2024-03-29T11:05:29.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.439+0000 7faa45b4d640 4 set_mon_vals no callback set 2024-03-29T11:05:29.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.439+0000 7faa45b4d640 10 monclient: shutdown 2024-03-29T11:05:29.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.439+0000 7faa45b4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:29.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.439+0000 7faa45b4d640 1 -- 172.21.15.71:0/765265544 >> v1:172.21.15.71:6790/0 conn(0x7faa400a9ac0 legacy=0x7faa400adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:29.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.440+0000 7faa45b4d640 1 -- 172.21.15.71:0/765265544 shutdown_connections 2024-03-29T11:05:29.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.440+0000 7faa45b4d640 1 -- 172.21.15.71:0/765265544 >> 172.21.15.71:0/765265544 conn(0x7faa400b35e0 msgr2=0x7faa400b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:29.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.441+0000 7faa45b4d640 1 -- 172.21.15.71:0/765265544 shutdown_connections 2024-03-29T11:05:29.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.442+0000 7faa45b4d640 1 -- 172.21.15.71:0/765265544 wait complete. 2024-03-29T11:05:29.426 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.442+0000 7faa45b4d640 10 monclient: build_initial_monmap 2024-03-29T11:05:29.426 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.442+0000 7faa45b4d640 10 monclient: monmap: 2024-03-29T11:05:29.426 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:05:29.426 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:05:29.426 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:05:29.426 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:05:29.426 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:29.426 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:29.426 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:05:29.426 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:05:29.426 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:05:29.426 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:29.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.443+0000 7faa45b4d640 1 librados: starting msgr at 2024-03-29T11:05:29.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.443+0000 7faa45b4d640 1 librados: starting objecter 2024-03-29T11:05:29.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.443+0000 7faa45b4d640 1 Processor -- start 2024-03-29T11:05:29.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.443+0000 7faa45b4d640 1 -- start start 2024-03-29T11:05:29.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.443+0000 7faa45b4d640 1 librados: setting wanted keys 2024-03-29T11:05:29.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.443+0000 7faa45b4d640 1 librados: calling monclient init 2024-03-29T11:05:29.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.443+0000 7faa45b4d640 10 monclient: init 2024-03-29T11:05:29.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.444+0000 7faa45b4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:29.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.444+0000 7faa45b4d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:05:29.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.444+0000 7faa45b4d640 10 monclient(hunting): picked mon.noname-a con 0x7faa400af900 addr v1:172.21.15.71:6789/0 2024-03-29T11:05:29.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.444+0000 7faa45b4d640 10 monclient(hunting): picked mon.noname-b con 0x7faa4013ee90 addr v1:172.21.15.137:6789/0 2024-03-29T11:05:29.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.444+0000 7faa45b4d640 10 monclient(hunting): picked mon.noname-c con 0x7faa401c5a80 addr v1:172.21.15.71:6790/0 2024-03-29T11:05:29.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.444+0000 7faa45b4d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa401c6bf0 con 0x7faa400af900 2024-03-29T11:05:29.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.444+0000 7faa45b4d640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa400718b0 con 0x7faa4013ee90 2024-03-29T11:05:29.428 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.444+0000 7faa45b4d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa40071a40 con 0x7faa401c5a80 2024-03-29T11:05:29.428 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.444+0000 7faa45b4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:29.428 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.444+0000 7faa45b4d640 10 monclient(hunting): authenticate will time out at 1085.702829s 2024-03-29T11:05:29.430 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.448+0000 7faa3ffff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7faa401c5a80 0x7faa401c8c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:49500/0 (socket says 172.21.15.71:49500) 2024-03-29T11:05:29.430 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.448+0000 7faa3ffff640 1 -- 172.21.15.71:0/3017741618 learned_addr learned my addr 172.21.15.71:0/3017741618 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:05:29.430 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.449+0000 7faa27fff640 10 client.?.objecter ms_handle_connect 0x7faa401c5a80 2024-03-29T11:05:29.430 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.449+0000 7faa27fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:29.430 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.449+0000 7faa27fff640 10 client.?.objecter ms_handle_connect 0x7faa400af900 2024-03-29T11:05:29.430 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.449+0000 7faa27fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:29.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.449+0000 7faa27fff640 1 -- 172.21.15.71:0/3017741618 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2446872602 0 0) 0x7faa401c6bf0 con 0x7faa400af900 2024-03-29T11:05:29.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.449+0000 7faa27fff640 10 monclient(hunting): _negotiate 2024-03-29T11:05:29.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.449+0000 7faa27fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:29.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.449+0000 7faa27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:29.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.449+0000 7faa27fff640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:05:29.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.450+0000 7faa27fff640 1 -- 172.21.15.71:0/3017741618 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa2c004eb0 con 0x7faa400af900 2024-03-29T11:05:29.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.450+0000 7faa27fff640 1 -- 172.21.15.71:0/3017741618 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4106227739 0 0) 0x7faa40071a40 con 0x7faa401c5a80 2024-03-29T11:05:29.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.450+0000 7faa27fff640 10 monclient(hunting): _negotiate 2024-03-29T11:05:29.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.450+0000 7faa27fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:29.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.450+0000 7faa27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:29.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.450+0000 7faa27fff640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:05:29.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.450+0000 7faa27fff640 1 -- 172.21.15.71:0/3017741618 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa401c6bf0 con 0x7faa401c5a80 2024-03-29T11:05:29.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.450+0000 7faa27fff640 1 -- 172.21.15.71:0/3017741618 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1915175714 0 0) 0x7faa2c004eb0 con 0x7faa400af900 2024-03-29T11:05:29.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.450+0000 7faa27fff640 1 -- 172.21.15.71:0/3017741618 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa40071a40 con 0x7faa400af900 2024-03-29T11:05:29.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.450+0000 7faa27fff640 1 -- 172.21.15.71:0/3017741618 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7faa30001550 con 0x7faa400af900 2024-03-29T11:05:29.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.450+0000 7faa27fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:29.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.450+0000 7faa27fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:05:29.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.450+0000 7faa27fff640 10 monclient(hunting): dump: 2024-03-29T11:05:29.432 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:29.432 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:29.432 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:29.432 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:29.432 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:29.432 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:29.432 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:29.432 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:29.432 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:29.432 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:29.433 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.450+0000 7faa27fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:05:29.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.451+0000 7faa27fff640 1 -- 172.21.15.71:0/3017741618 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1016418785 0 0) 0x7faa40071a40 con 0x7faa400af900 2024-03-29T11:05:29.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.451+0000 7faa27fff640 1 -- 172.21.15.71:0/3017741618 >> v1:172.21.15.71:6790/0 conn(0x7faa401c5a80 legacy=0x7faa401c8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:29.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.451+0000 7faa27fff640 1 -- 172.21.15.71:0/3017741618 >> v1:172.21.15.137:6789/0 conn(0x7faa4013ee90 legacy=0x7faa401c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:29.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.451+0000 7faa27fff640 10 monclient: _finish_hunting 0 2024-03-29T11:05:29.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.451+0000 7faa27fff640 1 monclient: found mon.a 2024-03-29T11:05:29.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.451+0000 7faa27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:29.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.451+0000 7faa27fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:05:29.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.451+0000 7faa27fff640 1 -- 172.21.15.71:0/3017741618 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa40071bd0 con 0x7faa400af900 2024-03-29T11:05:29.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.451+0000 7faa27fff640 10 monclient: _finish_auth 0 2024-03-29T11:05:29.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.451+0000 7faa27fff640 10 monclient: _check_auth_tickets 2024-03-29T11:05:29.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.451+0000 7faa27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:29.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.453+0000 7faa27fff640 1 -- 172.21.15.71:0/3017741618 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa300016d0 con 0x7faa400af900 2024-03-29T11:05:29.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.453+0000 7faa27fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:29.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.453+0000 7faa27fff640 1 -- 172.21.15.71:0/3017741618 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7faa30004d90 con 0x7faa400af900 2024-03-29T11:05:29.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.453+0000 7faa27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:29.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.453+0000 7faa27fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:05:29.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.453+0000 7faa27fff640 10 monclient: dump: 2024-03-29T11:05:29.436 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:29.436 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:29.436 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:29.436 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:29.436 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:29.436 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:29.436 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:29.436 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:29.436 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:29.436 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:29.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.453+0000 7faa3d7fa640 4 set_mon_vals no callback set 2024-03-29T11:05:29.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.453+0000 7faa45b4d640 5 monclient: authenticate success, global_id 4419 2024-03-29T11:05:29.437 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.453+0000 7faa45b4d640 10 monclient: _renew_subs 2024-03-29T11:05:29.437 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.453+0000 7faa45b4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:05:29.437 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.453+0000 7faa45b4d640 1 -- 172.21.15.71:0/3017741618 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa401e18a0 con 0x7faa400af900 2024-03-29T11:05:29.437 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.453+0000 7faa45b4d640 10 client.4419.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:05:29.437 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.453+0000 7faa45b4d640 10 monclient: _renew_subs 2024-03-29T11:05:29.437 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.453+0000 7faa45b4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:05:29.437 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.453+0000 7faa45b4d640 1 -- 172.21.15.71:0/3017741618 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa401c6490 con 0x7faa400af900 2024-03-29T11:05:29.437 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.453+0000 7faa45b4d640 1 librados: init done 2024-03-29T11:05:29.437 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.454+0000 7faa45b4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:29.437 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.455+0000 7faa45b4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:29.437 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.455+0000 7faa45b4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:05:29.437 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.455+0000 7faa45b4d640 1 -- 172.21.15.71:0/3017741618 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa401c7fc0 con 0x7faa400af900 2024-03-29T11:05:29.438 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.456+0000 7faa27fff640 1 -- 172.21.15.71:0/3017741618 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7faa3001ad60 con 0x7faa400af900 2024-03-29T11:05:29.438 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.457+0000 7faa27fff640 1 -- 172.21.15.71:0/3017741618 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7faa30084a60 con 0x7faa400af900 2024-03-29T11:05:29.438 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.457+0000 7faa27fff640 10 client.4419.objecter ms_dispatch 0x7faa40000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:05:29.438 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.457+0000 7faa27fff640 3 client.4419.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:05:29.438 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.457+0000 7faa27fff640 3 client.4419.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:05:29.438 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.457+0000 7faa27fff640 20 client.4419.objecter dump_active .. 0 homeless 2024-03-29T11:05:29.439 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.457+0000 7faa3effd640 10 monclient: get_auth_request con 0x7faa2c06ddf0 auth_method 0 2024-03-29T11:05:29.439 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.457+0000 7faa27fff640 10 client.4419.objecter ms_handle_connect 0x7faa2c06ddf0 2024-03-29T11:05:29.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.464+0000 7faa27fff640 1 -- 172.21.15.71:0/3017741618 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faa30050340 con 0x7faa400af900 2024-03-29T11:05:29.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.464+0000 7faa27fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:29.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.464+0000 7faa27fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:05:29.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.787+0000 7faa45b4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:29.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.787+0000 7faa45b4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:29.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.787+0000 7faa45b4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:05:29.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.787+0000 7faa45b4d640 1 -- 172.21.15.71:0/3017741618 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa401c6d80 con 0x7faa400af900 2024-03-29T11:05:29.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.790+0000 7faa27fff640 1 -- 172.21.15.71:0/3017741618 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7faa30050340 con 0x7faa400af900 2024-03-29T11:05:29.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.790+0000 7faa27fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:29.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.790+0000 7faa27fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:05:29.770 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:05:29.770 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:05:29.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.794+0000 7faa45b4d640 10 librados: watch_flush enter 2024-03-29T11:05:29.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.794+0000 7faa45b4d640 10 librados: watch_flush exit 2024-03-29T11:05:29.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.794+0000 7faa45b4d640 20 client.4419.objecter shutdown clearing up homeless session... 2024-03-29T11:05:29.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.794+0000 7faa45b4d640 10 client.4419.objecter successfully canceled tick 2024-03-29T11:05:29.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.794+0000 7faa45b4d640 1 -- 172.21.15.71:0/3017741618 >> v1:172.21.15.137:6816/973332665 conn(0x7faa2c06ddf0 legacy=0x7faa2c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:29.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.794+0000 7faa45b4d640 10 monclient: shutdown 2024-03-29T11:05:29.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.794+0000 7faa45b4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:29.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.794+0000 7faa45b4d640 1 -- 172.21.15.71:0/3017741618 >> v1:172.21.15.71:6789/0 conn(0x7faa400af900 legacy=0x7faa401e1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:29.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.794+0000 7faa45b4d640 1 -- 172.21.15.71:0/3017741618 shutdown_connections 2024-03-29T11:05:29.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.794+0000 7faa45b4d640 1 -- 172.21.15.71:0/3017741618 >> 172.21.15.71:0/3017741618 conn(0x7faa400b35e0 msgr2=0x7faa400ab760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:29.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.795+0000 7faa45b4d640 1 -- 172.21.15.71:0/3017741618 shutdown_connections 2024-03-29T11:05:29.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.795+0000 7faa45b4d640 1 -- 172.21.15.71:0/3017741618 wait complete. 2024-03-29T11:05:29.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:29.795+0000 7faa45b4d640 1 librados: shutdown 2024-03-29T11:05:29.791 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:05:36.598 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:05:36.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.762+0000 7f8b5bd65640 10 monclient: get_monmap_and_config 2024-03-29T11:05:36.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.762+0000 7f8b5bd65640 10 monclient: build_initial_monmap 2024-03-29T11:05:36.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.763+0000 7f8b5bd65640 10 monclient: monmap: 2024-03-29T11:05:36.742 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:05:36.742 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:36.742 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:05:36.763852+0000 2024-03-29T11:05:36.742 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:05:36.763852+0000 2024-03-29T11:05:36.742 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:36.743 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:36.743 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:05:36.743 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:05:36.743 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:05:36.743 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:36.743 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.764+0000 7f8b5bd65640 1 Processor -- start 2024-03-29T11:05:36.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.765+0000 7f8b5bd65640 1 -- start start 2024-03-29T11:05:36.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.765+0000 7f8b5bd65640 10 monclient: init 2024-03-29T11:05:36.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.765+0000 7f8b5bd65640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:36.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.765+0000 7f8b5bd65640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:05:36.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.765+0000 7f8b5bd65640 10 monclient(hunting): picked mon.noname-a con 0x7f8b541457f0 addr v1:172.21.15.71:6789/0 2024-03-29T11:05:36.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.765+0000 7f8b5bd65640 10 monclient(hunting): picked mon.noname-c con 0x7f8b5413f650 addr v1:172.21.15.137:6789/0 2024-03-29T11:05:36.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.765+0000 7f8b5bd65640 10 monclient(hunting): picked mon.noname-b con 0x7f8b540a9d70 addr v1:172.21.15.71:6790/0 2024-03-29T11:05:36.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.766+0000 7f8b5bd65640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b540aaef0 con 0x7f8b541457f0 2024-03-29T11:05:36.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.766+0000 7f8b5bd65640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b54145bd0 con 0x7f8b5413f650 2024-03-29T11:05:36.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.766+0000 7f8b5bd65640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b54145eb0 con 0x7f8b540a9d70 2024-03-29T11:05:36.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.766+0000 7f8b5bd65640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:36.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.766+0000 7f8b5bd65640 10 monclient(hunting): authenticate will time out at 1093.024138s 2024-03-29T11:05:36.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.766+0000 7f8b5a302640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f8b540a9d70 0x7f8b5414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:36820/0 (socket says 172.21.15.71:36820) 2024-03-29T11:05:36.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.766+0000 7f8b5a302640 1 -- 172.21.15.71:0/4126864513 learned_addr learned my addr 172.21.15.71:0/4126864513 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:05:36.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.767+0000 7f8b58aff640 1 -- 172.21.15.71:0/4126864513 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2474061112 0 0) 0x7f8b54145eb0 con 0x7f8b540a9d70 2024-03-29T11:05:36.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.767+0000 7f8b58aff640 10 monclient(hunting): _negotiate 2024-03-29T11:05:36.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.767+0000 7f8b58aff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:36.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.767+0000 7f8b58aff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:36.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.767+0000 7f8b58aff640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:05:36.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.767+0000 7f8b58aff640 1 -- 172.21.15.71:0/4126864513 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b40003520 con 0x7f8b540a9d70 2024-03-29T11:05:36.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.767+0000 7f8b58aff640 1 -- 172.21.15.71:0/4126864513 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2635151136 0 0) 0x7f8b40003520 con 0x7f8b540a9d70 2024-03-29T11:05:36.747 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.767+0000 7f8b58aff640 1 -- 172.21.15.71:0/4126864513 >> v1:172.21.15.137:6789/0 conn(0x7f8b5413f650 legacy=0x7f8b540b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:05:36.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.769+0000 7f8b58aff640 1 -- 172.21.15.71:0/4126864513 >> v1:172.21.15.71:6789/0 conn(0x7f8b541457f0 legacy=0x7f8b5413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:05:36.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.769+0000 7f8b58aff640 10 monclient: _finish_hunting 0 2024-03-29T11:05:36.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.769+0000 7f8b58aff640 1 monclient: found mon.noname-b 2024-03-29T11:05:36.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.769+0000 7f8b58aff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:36.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.769+0000 7f8b58aff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:05:36.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.769+0000 7f8b58aff640 1 -- 172.21.15.71:0/4126864513 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b54146340 con 0x7f8b540a9d70 2024-03-29T11:05:36.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.769+0000 7f8b58aff640 10 monclient: _finish_auth 0 2024-03-29T11:05:36.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.769+0000 7f8b58aff640 10 monclient: _check_auth_tickets 2024-03-29T11:05:36.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.769+0000 7f8b58aff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:36.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.769+0000 7f8b58aff640 1 -- 172.21.15.71:0/4126864513 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f8b50001690 con 0x7f8b540a9d70 2024-03-29T11:05:36.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.769+0000 7f8b58aff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:36.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.769+0000 7f8b58aff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:05:36.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.769+0000 7f8b58aff640 10 monclient: dump: 2024-03-29T11:05:36.751 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:36.751 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:36.751 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:36.751 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:36.751 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:36.751 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:36.751 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:36.751 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:36.752 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:36.752 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:36.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.769+0000 7f8b5bd65640 5 monclient: authenticate success, global_id 4445 2024-03-29T11:05:36.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.769+0000 7f8b5bd65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:05:36.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.771+0000 7f8b58aff640 1 -- 172.21.15.71:0/4126864513 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b50001860 con 0x7f8b540a9d70 2024-03-29T11:05:36.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.771+0000 7f8b58aff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:36.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.771+0000 7f8b58aff640 1 -- 172.21.15.71:0/4126864513 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f8b50001cb0 con 0x7f8b540a9d70 2024-03-29T11:05:36.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.771+0000 7f8b58aff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:36.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.771+0000 7f8b58aff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:05:36.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.771+0000 7f8b58aff640 10 monclient: dump: 2024-03-29T11:05:36.752 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:36.752 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:36.752 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:36.752 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:36.752 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:36.752 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:36.753 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:36.753 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:36.753 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:36.753 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:36.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.771+0000 7f8b5bd65640 10 monclient: get_monmap_and_config success 2024-03-29T11:05:36.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.771+0000 7f8b5bd65640 4 set_mon_vals no callback set 2024-03-29T11:05:36.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.772+0000 7f8b5bd65640 10 monclient: shutdown 2024-03-29T11:05:36.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.772+0000 7f8b5bd65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:36.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.772+0000 7f8b5bd65640 1 -- 172.21.15.71:0/4126864513 >> v1:172.21.15.71:6790/0 conn(0x7f8b540a9d70 legacy=0x7f8b5414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:36.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.772+0000 7f8b5bd65640 1 -- 172.21.15.71:0/4126864513 shutdown_connections 2024-03-29T11:05:36.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.772+0000 7f8b5bd65640 1 -- 172.21.15.71:0/4126864513 >> 172.21.15.71:0/4126864513 conn(0x7f8b54066a00 msgr2=0x7f8b54066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:36.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.774+0000 7f8b5bd65640 1 -- 172.21.15.71:0/4126864513 shutdown_connections 2024-03-29T11:05:36.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.774+0000 7f8b5bd65640 1 -- 172.21.15.71:0/4126864513 wait complete. 2024-03-29T11:05:36.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.775+0000 7f8b5bd65640 10 monclient: build_initial_monmap 2024-03-29T11:05:36.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.775+0000 7f8b5bd65640 10 monclient: monmap: 2024-03-29T11:05:36.755 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:05:36.755 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:05:36.755 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:05:36.755 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:05:36.755 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:36.755 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:36.755 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:05:36.755 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:05:36.755 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:05:36.755 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:36.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.776+0000 7f8b5bd65640 1 librados: starting msgr at 2024-03-29T11:05:36.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.776+0000 7f8b5bd65640 1 librados: starting objecter 2024-03-29T11:05:36.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.776+0000 7f8b5bd65640 1 Processor -- start 2024-03-29T11:05:36.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.779+0000 7f8b5bd65640 1 -- start start 2024-03-29T11:05:36.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.779+0000 7f8b5bd65640 1 librados: setting wanted keys 2024-03-29T11:05:36.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.779+0000 7f8b5bd65640 1 librados: calling monclient init 2024-03-29T11:05:36.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.779+0000 7f8b5bd65640 10 monclient: init 2024-03-29T11:05:36.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.780+0000 7f8b5bd65640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:36.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.780+0000 7f8b5bd65640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:05:36.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.780+0000 7f8b5bd65640 10 monclient(hunting): picked mon.noname-c con 0x7f8b540a9d70 addr v1:172.21.15.71:6790/0 2024-03-29T11:05:36.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.780+0000 7f8b5bd65640 10 monclient(hunting): picked mon.noname-a con 0x7f8b5413f650 addr v1:172.21.15.71:6789/0 2024-03-29T11:05:36.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.780+0000 7f8b5bd65640 10 monclient(hunting): picked mon.noname-b con 0x7f8b541457f0 addr v1:172.21.15.137:6789/0 2024-03-29T11:05:36.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.780+0000 7f8b5bd65640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b54070100 con 0x7f8b5413f650 2024-03-29T11:05:36.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.780+0000 7f8b5bd65640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b5406de90 con 0x7f8b541457f0 2024-03-29T11:05:36.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.782+0000 7f8b59b01640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f8b540a9d70 0x7f8b54071650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:40680/0 (socket says 172.21.15.71:40680) 2024-03-29T11:05:36.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.782+0000 7f8b59b01640 1 -- 172.21.15.71:0/1956204569 learned_addr learned my addr 172.21.15.71:0/1956204569 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:05:36.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.782+0000 7f8b5bd65640 1 -- 172.21.15.71:0/1956204569 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b5406e0e0 con 0x7f8b540a9d70 2024-03-29T11:05:36.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.782+0000 7f8b5bd65640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:36.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.782+0000 7f8b5bd65640 10 monclient(hunting): authenticate will time out at 1093.040501s 2024-03-29T11:05:36.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.784+0000 7f8b4a7fc640 10 client.?.objecter ms_handle_connect 0x7f8b540a9d70 2024-03-29T11:05:36.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.784+0000 7f8b4a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:36.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.784+0000 7f8b4a7fc640 1 -- 172.21.15.71:0/1956204569 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727941300 0 0) 0x7f8b5406e0e0 con 0x7f8b540a9d70 2024-03-29T11:05:36.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.784+0000 7f8b4a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:05:36.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.784+0000 7f8b4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:36.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.784+0000 7f8b4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:36.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.784+0000 7f8b4a7fc640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:05:36.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.785+0000 7f8b4a7fc640 1 -- 172.21.15.71:0/1956204569 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b38004eb0 con 0x7f8b540a9d70 2024-03-29T11:05:36.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.785+0000 7f8b4a7fc640 10 client.?.objecter ms_handle_connect 0x7f8b541457f0 2024-03-29T11:05:36.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.785+0000 7f8b4a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:36.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.785+0000 7f8b4a7fc640 1 -- 172.21.15.71:0/1956204569 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3904884273 0 0) 0x7f8b38004eb0 con 0x7f8b540a9d70 2024-03-29T11:05:36.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.785+0000 7f8b4a7fc640 1 -- 172.21.15.71:0/1956204569 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b5406e0e0 con 0x7f8b540a9d70 2024-03-29T11:05:36.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.785+0000 7f8b4a7fc640 1 -- 172.21.15.71:0/1956204569 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f8b4c004360 con 0x7f8b540a9d70 2024-03-29T11:05:36.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.785+0000 7f8b4a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:36.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.785+0000 7f8b4a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:05:36.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.785+0000 7f8b4a7fc640 10 monclient(hunting): dump: 2024-03-29T11:05:36.765 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:36.765 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:36.765 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:36.765 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:36.765 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:36.765 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:36.765 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:36.765 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:36.765 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:36.765 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:36.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.785+0000 7f8b4a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:05:36.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.785+0000 7f8b4a7fc640 1 -- 172.21.15.71:0/1956204569 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1253320921 0 0) 0x7f8b5406de90 con 0x7f8b541457f0 2024-03-29T11:05:36.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.785+0000 7f8b4a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:05:36.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.785+0000 7f8b4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:36.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.785+0000 7f8b4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:36.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.785+0000 7f8b4a7fc640 10 monclient(hunting): my global_id is 4387 2024-03-29T11:05:36.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.785+0000 7f8b4a7fc640 1 -- 172.21.15.71:0/1956204569 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b38004eb0 con 0x7f8b541457f0 2024-03-29T11:05:36.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.787+0000 7f8b4a7fc640 1 -- 172.21.15.71:0/1956204569 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 973335489 0 0) 0x7f8b5406e0e0 con 0x7f8b540a9d70 2024-03-29T11:05:36.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.787+0000 7f8b4a7fc640 1 -- 172.21.15.71:0/1956204569 >> v1:172.21.15.137:6789/0 conn(0x7f8b541457f0 legacy=0x7f8b54070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:36.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.787+0000 7f8b4a7fc640 1 -- 172.21.15.71:0/1956204569 >> v1:172.21.15.71:6789/0 conn(0x7f8b5413f650 legacy=0x7f8b54140db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:05:36.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.787+0000 7f8b4a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:05:36.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.787+0000 7f8b4a7fc640 1 monclient: found mon.c 2024-03-29T11:05:36.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.787+0000 7f8b4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:36.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.787+0000 7f8b4a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:05:36.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.787+0000 7f8b4a7fc640 1 -- 172.21.15.71:0/1956204569 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b5406f720 con 0x7f8b540a9d70 2024-03-29T11:05:36.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.788+0000 7f8b4a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:05:36.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.788+0000 7f8b4a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:05:36.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.788+0000 7f8b4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:36.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.788+0000 7f8b5bd65640 5 monclient: authenticate success, global_id 4448 2024-03-29T11:05:36.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.788+0000 7f8b5bd65640 10 monclient: _renew_subs 2024-03-29T11:05:36.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.788+0000 7f8b5bd65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:05:36.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.788+0000 7f8b5bd65640 1 -- 172.21.15.71:0/1956204569 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b5406fa00 con 0x7f8b540a9d70 2024-03-29T11:05:36.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.788+0000 7f8b5bd65640 10 client.4448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:05:36.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.788+0000 7f8b5bd65640 10 monclient: _renew_subs 2024-03-29T11:05:36.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.788+0000 7f8b5bd65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:05:36.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.788+0000 7f8b5bd65640 1 -- 172.21.15.71:0/1956204569 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b54134700 con 0x7f8b540a9d70 2024-03-29T11:05:36.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.788+0000 7f8b5bd65640 1 librados: init done 2024-03-29T11:05:36.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.788+0000 7f8b4a7fc640 1 -- 172.21.15.71:0/1956204569 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b4c001250 con 0x7f8b540a9d70 2024-03-29T11:05:36.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.788+0000 7f8b4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:36.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.788+0000 7f8b4a7fc640 1 -- 172.21.15.71:0/1956204569 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f8b4c0044c0 con 0x7f8b540a9d70 2024-03-29T11:05:36.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.788+0000 7f8b4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:36.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.788+0000 7f8b4a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:05:36.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.788+0000 7f8b4b7fe640 4 set_mon_vals no callback set 2024-03-29T11:05:36.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.790+0000 7f8b4a7fc640 10 monclient: dump: 2024-03-29T11:05:36.771 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:36.771 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:36.771 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:36.771 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:36.772 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:36.772 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:36.772 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:36.772 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:36.772 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:36.772 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:36.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.790+0000 7f8b4a7fc640 1 -- 172.21.15.71:0/1956204569 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f8b4c001720 con 0x7f8b540a9d70 2024-03-29T11:05:36.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.790+0000 7f8b4a7fc640 1 -- 172.21.15.71:0/1956204569 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f8b4c084e40 con 0x7f8b540a9d70 2024-03-29T11:05:36.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.791+0000 7f8b4a7fc640 10 client.4448.objecter ms_dispatch 0x7f8b54000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:05:36.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.791+0000 7f8b4a7fc640 3 client.4448.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:05:36.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.791+0000 7f8b4a7fc640 3 client.4448.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:05:36.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.791+0000 7f8b4a7fc640 20 client.4448.objecter dump_active .. 0 homeless 2024-03-29T11:05:36.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.791+0000 7f8b59300640 10 monclient: get_auth_request con 0x7f8b3806dd50 auth_method 0 2024-03-29T11:05:36.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.791+0000 7f8b5bd65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:36.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.791+0000 7f8b5bd65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:36.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.791+0000 7f8b5bd65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:05:36.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.791+0000 7f8b5bd65640 1 -- 172.21.15.71:0/1956204569 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b3c005080 con 0x7f8b540a9d70 2024-03-29T11:05:36.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.793+0000 7f8b4a7fc640 10 client.4448.objecter ms_handle_connect 0x7f8b3806dd50 2024-03-29T11:05:36.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.801+0000 7f8b4a7fc640 1 -- 172.21.15.71:0/1956204569 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8b4c050720 con 0x7f8b540a9d70 2024-03-29T11:05:36.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.801+0000 7f8b4a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:36.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:36.801+0000 7f8b4a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:05:37.102 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:37.121+0000 7f8b5bd65640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:37.102 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:37.121+0000 7f8b5bd65640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:37.102 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:37.121+0000 7f8b5bd65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:05:37.102 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:37.121+0000 7f8b5bd65640 1 -- 172.21.15.71:0/1956204569 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b3c005330 con 0x7f8b540a9d70 2024-03-29T11:05:37.102 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:37.122+0000 7f8b4a7fc640 1 -- 172.21.15.71:0/1956204569 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f8b4c052650 con 0x7f8b540a9d70 2024-03-29T11:05:37.102 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:37.122+0000 7f8b4a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:37.102 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:37.122+0000 7f8b4a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:05:37.103 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:05:37.103 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:05:37.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:37.127+0000 7f8b5bd65640 10 librados: watch_flush enter 2024-03-29T11:05:37.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:37.127+0000 7f8b5bd65640 10 librados: watch_flush exit 2024-03-29T11:05:37.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:37.127+0000 7f8b5bd65640 20 client.4448.objecter shutdown clearing up homeless session... 2024-03-29T11:05:37.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:37.127+0000 7f8b5bd65640 10 client.4448.objecter successfully canceled tick 2024-03-29T11:05:37.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:37.127+0000 7f8b5bd65640 1 -- 172.21.15.71:0/1956204569 >> v1:172.21.15.137:6816/973332665 conn(0x7f8b3806dd50 legacy=0x7f8b38070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:37.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:37.127+0000 7f8b5bd65640 10 monclient: shutdown 2024-03-29T11:05:37.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:37.127+0000 7f8b5bd65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:37.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:37.127+0000 7f8b5bd65640 1 -- 172.21.15.71:0/1956204569 >> v1:172.21.15.71:6790/0 conn(0x7f8b540a9d70 legacy=0x7f8b54071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:37.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:37.127+0000 7f8b5bd65640 1 -- 172.21.15.71:0/1956204569 shutdown_connections 2024-03-29T11:05:37.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:37.127+0000 7f8b5bd65640 1 -- 172.21.15.71:0/1956204569 >> 172.21.15.71:0/1956204569 conn(0x7f8b54066a00 msgr2=0x7f8b540b60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:37.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:37.128+0000 7f8b5bd65640 1 -- 172.21.15.71:0/1956204569 shutdown_connections 2024-03-29T11:05:37.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:37.128+0000 7f8b5bd65640 1 -- 172.21.15.71:0/1956204569 wait complete. 2024-03-29T11:05:37.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:37.128+0000 7f8b5bd65640 1 librados: shutdown 2024-03-29T11:05:37.123 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:05:43.930 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:05:44.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.102+0000 7f4a70951640 10 monclient: get_monmap_and_config 2024-03-29T11:05:44.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.102+0000 7f4a70951640 10 monclient: build_initial_monmap 2024-03-29T11:05:44.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.102+0000 7f4a70951640 10 monclient: monmap: 2024-03-29T11:05:44.086 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:05:44.086 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:44.086 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:05:44.104140+0000 2024-03-29T11:05:44.087 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:05:44.104140+0000 2024-03-29T11:05:44.087 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:44.087 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:44.087 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:05:44.087 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:05:44.087 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:05:44.087 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:44.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.104+0000 7f4a70951640 1 Processor -- start 2024-03-29T11:05:44.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.107+0000 7f4a70951640 1 -- start start 2024-03-29T11:05:44.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.107+0000 7f4a70951640 10 monclient: init 2024-03-29T11:05:44.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.108+0000 7f4a70951640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:44.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.108+0000 7f4a70951640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:05:44.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.108+0000 7f4a70951640 10 monclient(hunting): picked mon.noname-c con 0x7f4a64006410 addr v1:172.21.15.137:6789/0 2024-03-29T11:05:44.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.108+0000 7f4a70951640 10 monclient(hunting): picked mon.noname-a con 0x7f4a6402a640 addr v1:172.21.15.71:6789/0 2024-03-29T11:05:44.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.108+0000 7f4a70951640 10 monclient(hunting): picked mon.noname-b con 0x7f4a640327d0 addr v1:172.21.15.71:6790/0 2024-03-29T11:05:44.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.108+0000 7f4a70951640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a6402c820 con 0x7f4a6402a640 2024-03-29T11:05:44.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.108+0000 7f4a70951640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a6402a120 con 0x7f4a64006410 2024-03-29T11:05:44.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.108+0000 7f4a70951640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a640c75b0 con 0x7f4a640327d0 2024-03-29T11:05:44.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.108+0000 7f4a70951640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:44.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.108+0000 7f4a70951640 10 monclient(hunting): authenticate will time out at 1100.367246s 2024-03-29T11:05:44.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.110+0000 7f4a6ad9d640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f4a64006410 0x7f4a640c6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:48505/0 (socket says 172.21.15.71:48505) 2024-03-29T11:05:44.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.110+0000 7f4a6ad9d640 1 -- 172.21.15.71:0/3861429375 learned_addr learned my addr 172.21.15.71:0/3861429375 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:05:44.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.112+0000 7f4a69d9b640 1 -- 172.21.15.71:0/3861429375 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572023813 0 0) 0x7f4a6402a120 con 0x7f4a64006410 2024-03-29T11:05:44.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.112+0000 7f4a69d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:05:44.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.112+0000 7f4a69d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:44.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.112+0000 7f4a69d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:44.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.112+0000 7f4a69d9b640 10 monclient(hunting): my global_id is 4390 2024-03-29T11:05:44.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.112+0000 7f4a69d9b640 1 -- 172.21.15.71:0/3861429375 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a54003520 con 0x7f4a64006410 2024-03-29T11:05:44.094 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.112+0000 7f4a69d9b640 1 -- 172.21.15.71:0/3861429375 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1434888833 0 0) 0x7f4a54003520 con 0x7f4a64006410 2024-03-29T11:05:44.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.112+0000 7f4a69d9b640 1 -- 172.21.15.71:0/3861429375 >> v1:172.21.15.71:6790/0 conn(0x7f4a640327d0 legacy=0x7f4a640c8200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:05:44.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.113+0000 7f4a69d9b640 1 -- 172.21.15.71:0/3861429375 >> v1:172.21.15.71:6789/0 conn(0x7f4a6402a640 legacy=0x7f4a640320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:44.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.113+0000 7f4a69d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:05:44.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.113+0000 7f4a69d9b640 1 monclient: found mon.noname-c 2024-03-29T11:05:44.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.113+0000 7f4a69d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:44.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.113+0000 7f4a69d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.137:6789/0 2024-03-29T11:05:44.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.113+0000 7f4a69d9b640 1 -- 172.21.15.71:0/3861429375 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a64033110 con 0x7f4a64006410 2024-03-29T11:05:44.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.113+0000 7f4a69d9b640 10 monclient: _finish_auth 0 2024-03-29T11:05:44.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.113+0000 7f4a69d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:05:44.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.113+0000 7f4a69d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:44.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.113+0000 7f4a70951640 5 monclient: authenticate success, global_id 4390 2024-03-29T11:05:44.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.113+0000 7f4a70951640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:05:44.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.113+0000 7f4a69d9b640 1 -- 172.21.15.71:0/3861429375 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4a60001690 con 0x7f4a64006410 2024-03-29T11:05:44.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.113+0000 7f4a69d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:44.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.113+0000 7f4a69d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:05:44.095 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.113+0000 7f4a69d9b640 10 monclient: dump: 2024-03-29T11:05:44.096 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:44.096 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:44.096 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:44.096 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:44.096 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:44.096 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:44.096 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:44.096 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:44.096 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:44.096 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:44.096 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.113+0000 7f4a70951640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:05:44.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.114+0000 7f4a69d9b640 1 -- 172.21.15.71:0/3861429375 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a60001920 con 0x7f4a64006410 2024-03-29T11:05:44.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.114+0000 7f4a69d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:44.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.114+0000 7f4a70951640 10 monclient: get_monmap_and_config success 2024-03-29T11:05:44.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.114+0000 7f4a70951640 4 set_mon_vals no callback set 2024-03-29T11:05:44.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.115+0000 7f4a69d9b640 1 -- 172.21.15.71:0/3861429375 <== mon.1 v1:172.21.15.137:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4a60001cf0 con 0x7f4a64006410 2024-03-29T11:05:44.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.115+0000 7f4a70951640 10 monclient: shutdown 2024-03-29T11:05:44.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.115+0000 7f4a69d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:44.099 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.115+0000 7f4a69d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:05:44.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.115+0000 7f4a69d9b640 10 monclient: dump: 2024-03-29T11:05:44.100 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:44.100 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:44.100 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:44.100 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:44.100 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:44.100 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:44.100 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:44.100 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:44.100 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:44.100 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:44.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.115+0000 7f4a70951640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:44.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.115+0000 7f4a70951640 1 -- 172.21.15.71:0/3861429375 >> v1:172.21.15.137:6789/0 conn(0x7f4a64006410 legacy=0x7f4a640c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:44.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.116+0000 7f4a70951640 1 -- 172.21.15.71:0/3861429375 shutdown_connections 2024-03-29T11:05:44.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.116+0000 7f4a70951640 1 -- 172.21.15.71:0/3861429375 >> 172.21.15.71:0/3861429375 conn(0x7f4a640058f0 msgr2=0x7f4a64005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:44.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.116+0000 7f4a70951640 1 -- 172.21.15.71:0/3861429375 shutdown_connections 2024-03-29T11:05:44.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.116+0000 7f4a70951640 1 -- 172.21.15.71:0/3861429375 wait complete. 2024-03-29T11:05:44.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.118+0000 7f4a70951640 10 monclient: build_initial_monmap 2024-03-29T11:05:44.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.118+0000 7f4a70951640 10 monclient: monmap: 2024-03-29T11:05:44.101 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:05:44.101 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:05:44.101 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:05:44.101 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:05:44.101 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:44.101 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:44.102 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:05:44.102 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:05:44.102 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:05:44.102 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:44.102 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.118+0000 7f4a70951640 1 librados: starting msgr at 2024-03-29T11:05:44.102 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.118+0000 7f4a70951640 1 librados: starting objecter 2024-03-29T11:05:44.102 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.118+0000 7f4a70951640 1 Processor -- start 2024-03-29T11:05:44.102 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.119+0000 7f4a70951640 1 -- start start 2024-03-29T11:05:44.102 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.119+0000 7f4a70951640 1 librados: setting wanted keys 2024-03-29T11:05:44.102 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.119+0000 7f4a70951640 1 librados: calling monclient init 2024-03-29T11:05:44.103 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.119+0000 7f4a70951640 10 monclient: init 2024-03-29T11:05:44.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.120+0000 7f4a70951640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:44.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.120+0000 7f4a70951640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:05:44.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.120+0000 7f4a70951640 10 monclient(hunting): picked mon.noname-a con 0x7f4a64006410 addr v1:172.21.15.71:6789/0 2024-03-29T11:05:44.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.120+0000 7f4a70951640 10 monclient(hunting): picked mon.noname-b con 0x7f4a6402a640 addr v1:172.21.15.137:6789/0 2024-03-29T11:05:44.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.120+0000 7f4a70951640 10 monclient(hunting): picked mon.noname-c con 0x7f4a640327d0 addr v1:172.21.15.71:6790/0 2024-03-29T11:05:44.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.120+0000 7f4a70951640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a6402d900 con 0x7f4a64006410 2024-03-29T11:05:44.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.120+0000 7f4a70951640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a6402db50 con 0x7f4a6402a640 2024-03-29T11:05:44.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.120+0000 7f4a70951640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a6402de30 con 0x7f4a640327d0 2024-03-29T11:05:44.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.120+0000 7f4a70951640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:44.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.120+0000 7f4a70951640 10 monclient(hunting): authenticate will time out at 1100.379208s 2024-03-29T11:05:44.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.123+0000 7f4a6a59c640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f4a6402a640 0x7f4a640c0760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:48591/0 (socket says 172.21.15.71:48591) 2024-03-29T11:05:44.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.123+0000 7f4a6a59c640 1 -- 172.21.15.71:0/1591292959 learned_addr learned my addr 172.21.15.71:0/1591292959 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:05:44.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.126+0000 7f4a537fe640 10 client.?.objecter ms_handle_connect 0x7f4a6402a640 2024-03-29T11:05:44.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.126+0000 7f4a537fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:44.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.126+0000 7f4a537fe640 1 -- 172.21.15.71:0/1591292959 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2833750927 0 0) 0x7f4a6402db50 con 0x7f4a6402a640 2024-03-29T11:05:44.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.126+0000 7f4a537fe640 10 monclient(hunting): _negotiate 2024-03-29T11:05:44.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.126+0000 7f4a537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:44.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.126+0000 7f4a537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:44.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.126+0000 7f4a537fe640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:05:44.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.126+0000 7f4a537fe640 1 -- 172.21.15.71:0/1591292959 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a58004eb0 con 0x7f4a6402a640 2024-03-29T11:05:44.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.127+0000 7f4a537fe640 10 client.?.objecter ms_handle_connect 0x7f4a64006410 2024-03-29T11:05:44.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.127+0000 7f4a537fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:44.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.127+0000 7f4a537fe640 1 -- 172.21.15.71:0/1591292959 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2891067428 0 0) 0x7f4a58004eb0 con 0x7f4a6402a640 2024-03-29T11:05:44.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.127+0000 7f4a537fe640 1 -- 172.21.15.71:0/1591292959 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a6402db50 con 0x7f4a6402a640 2024-03-29T11:05:44.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.127+0000 7f4a537fe640 1 -- 172.21.15.71:0/1591292959 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306500494 0 0) 0x7f4a6402d900 con 0x7f4a64006410 2024-03-29T11:05:44.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.127+0000 7f4a537fe640 10 monclient(hunting): _negotiate 2024-03-29T11:05:44.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.127+0000 7f4a537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:44.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.127+0000 7f4a537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:44.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.127+0000 7f4a537fe640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:05:44.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.127+0000 7f4a537fe640 1 -- 172.21.15.71:0/1591292959 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a58004eb0 con 0x7f4a64006410 2024-03-29T11:05:44.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.127+0000 7f4a537fe640 1 -- 172.21.15.71:0/1591292959 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4a5c001960 con 0x7f4a6402a640 2024-03-29T11:05:44.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.127+0000 7f4a537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:44.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.127+0000 7f4a537fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:05:44.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.127+0000 7f4a537fe640 10 monclient(hunting): dump: 2024-03-29T11:05:44.109 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:44.109 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:44.109 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:44.109 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:44.109 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:44.109 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:44.109 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:44.110 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:44.110 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:44.110 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:44.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.127+0000 7f4a537fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:05:44.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.128+0000 7f4a537fe640 1 -- 172.21.15.71:0/1591292959 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1155525933 0 0) 0x7f4a58004eb0 con 0x7f4a64006410 2024-03-29T11:05:44.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.128+0000 7f4a537fe640 1 -- 172.21.15.71:0/1591292959 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a6402d900 con 0x7f4a64006410 2024-03-29T11:05:44.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.128+0000 7f4a537fe640 1 -- 172.21.15.71:0/1591292959 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4a600014b0 con 0x7f4a64006410 2024-03-29T11:05:44.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.128+0000 7f4a537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:44.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.128+0000 7f4a537fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:05:44.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.128+0000 7f4a537fe640 10 monclient(hunting): dump: 2024-03-29T11:05:44.110 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:44.110 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:44.110 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:44.110 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:44.111 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:44.111 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:44.111 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:44.111 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:44.111 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:44.111 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:44.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.128+0000 7f4a537fe640 1 -- 172.21.15.71:0/1591292959 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 876753235 0 0) 0x7f4a6402d900 con 0x7f4a64006410 2024-03-29T11:05:44.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.128+0000 7f4a537fe640 1 -- 172.21.15.71:0/1591292959 >> v1:172.21.15.71:6790/0 conn(0x7f4a640327d0 legacy=0x7f4a6416bb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:05:44.112 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.129+0000 7f4a537fe640 1 -- 172.21.15.71:0/1591292959 >> v1:172.21.15.137:6789/0 conn(0x7f4a6402a640 legacy=0x7f4a640c0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:44.112 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.129+0000 7f4a537fe640 10 monclient: _finish_hunting 0 2024-03-29T11:05:44.112 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.129+0000 7f4a537fe640 1 monclient: found mon.a 2024-03-29T11:05:44.112 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.129+0000 7f4a537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:44.112 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.129+0000 7f4a537fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:05:44.112 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.129+0000 7f4a537fe640 1 -- 172.21.15.71:0/1591292959 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a640312b0 con 0x7f4a64006410 2024-03-29T11:05:44.112 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.129+0000 7f4a537fe640 10 monclient: _finish_auth 0 2024-03-29T11:05:44.112 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.129+0000 7f4a537fe640 10 monclient: _check_auth_tickets 2024-03-29T11:05:44.112 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.129+0000 7f4a537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:44.112 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.129+0000 7f4a70951640 5 monclient: authenticate success, global_id 4425 2024-03-29T11:05:44.112 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.129+0000 7f4a70951640 10 monclient: _renew_subs 2024-03-29T11:05:44.112 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.129+0000 7f4a70951640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:05:44.112 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.129+0000 7f4a70951640 1 -- 172.21.15.71:0/1591292959 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a6402cc40 con 0x7f4a64006410 2024-03-29T11:05:44.113 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.129+0000 7f4a70951640 10 client.4425.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:05:44.113 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.129+0000 7f4a70951640 10 monclient: _renew_subs 2024-03-29T11:05:44.113 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.129+0000 7f4a70951640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:05:44.113 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.129+0000 7f4a70951640 1 -- 172.21.15.71:0/1591292959 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a640c46c0 con 0x7f4a64006410 2024-03-29T11:05:44.113 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.129+0000 7f4a70951640 1 librados: init done 2024-03-29T11:05:44.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.133+0000 7f4a70951640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:44.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.133+0000 7f4a537fe640 1 -- 172.21.15.71:0/1591292959 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a60004980 con 0x7f4a64006410 2024-03-29T11:05:44.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.133+0000 7f4a70951640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:44.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.133+0000 7f4a70951640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:05:44.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.133+0000 7f4a70951640 1 -- 172.21.15.71:0/1591292959 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a640c7380 con 0x7f4a64006410 2024-03-29T11:05:44.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.133+0000 7f4a537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:44.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.133+0000 7f4a537fe640 1 -- 172.21.15.71:0/1591292959 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4a60004c90 con 0x7f4a64006410 2024-03-29T11:05:44.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.133+0000 7f4a537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:44.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.133+0000 7f4a537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:05:44.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.133+0000 7f4a537fe640 10 monclient: dump: 2024-03-29T11:05:44.115 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:44.115 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:44.115 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:44.115 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:44.115 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:44.115 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:44.115 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:44.115 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:44.115 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:44.115 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:44.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.134+0000 7f4a537fe640 1 -- 172.21.15.71:0/1591292959 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f4a6001ac60 con 0x7f4a64006410 2024-03-29T11:05:44.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.135+0000 7f4a537fe640 1 -- 172.21.15.71:0/1591292959 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f4a60084a20 con 0x7f4a64006410 2024-03-29T11:05:44.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.135+0000 7f4a537fe640 10 client.4425.objecter ms_dispatch 0x7f4a6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:05:44.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.135+0000 7f4a537fe640 3 client.4425.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:05:44.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.135+0000 7f4a537fe640 3 client.4425.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:05:44.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.135+0000 7f4a537fe640 20 client.4425.objecter dump_active .. 0 homeless 2024-03-29T11:05:44.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.134+0000 7f4a6959a640 4 set_mon_vals no callback set 2024-03-29T11:05:44.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.137+0000 7f4a6a59c640 10 monclient: get_auth_request con 0x7f4a5806df60 auth_method 0 2024-03-29T11:05:44.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.137+0000 7f4a537fe640 10 client.4425.objecter ms_handle_connect 0x7f4a5806df60 2024-03-29T11:05:44.126 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.146+0000 7f4a537fe640 1 -- 172.21.15.71:0/1591292959 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4a60050360 con 0x7f4a64006410 2024-03-29T11:05:44.126 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.146+0000 7f4a537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:44.126 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.146+0000 7f4a537fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:05:44.461 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.481+0000 7f4a70951640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:44.461 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.481+0000 7f4a70951640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:44.461 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.481+0000 7f4a70951640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:05:44.461 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.481+0000 7f4a70951640 1 -- 172.21.15.71:0/1591292959 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a64004cd0 con 0x7f4a64006410 2024-03-29T11:05:44.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.484+0000 7f4a537fe640 1 -- 172.21.15.71:0/1591292959 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f4a60052290 con 0x7f4a64006410 2024-03-29T11:05:44.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.484+0000 7f4a537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:44.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.484+0000 7f4a537fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:05:44.465 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:05:44.465 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:05:44.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.488+0000 7f4a70951640 10 librados: watch_flush enter 2024-03-29T11:05:44.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.488+0000 7f4a70951640 10 librados: watch_flush exit 2024-03-29T11:05:44.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.488+0000 7f4a70951640 20 client.4425.objecter shutdown clearing up homeless session... 2024-03-29T11:05:44.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.488+0000 7f4a70951640 10 client.4425.objecter successfully canceled tick 2024-03-29T11:05:44.469 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.489+0000 7f4a70951640 1 -- 172.21.15.71:0/1591292959 >> v1:172.21.15.137:6816/973332665 conn(0x7f4a5806df60 legacy=0x7f4a58070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:44.469 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.489+0000 7f4a70951640 10 monclient: shutdown 2024-03-29T11:05:44.469 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.489+0000 7f4a70951640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:44.469 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.489+0000 7f4a70951640 1 -- 172.21.15.71:0/1591292959 >> v1:172.21.15.71:6789/0 conn(0x7f4a64006410 legacy=0x7f4a640c0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:44.469 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.489+0000 7f4a70951640 1 -- 172.21.15.71:0/1591292959 shutdown_connections 2024-03-29T11:05:44.469 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.489+0000 7f4a70951640 1 -- 172.21.15.71:0/1591292959 >> 172.21.15.71:0/1591292959 conn(0x7f4a640058f0 msgr2=0x7f4a64005cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:44.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.492+0000 7f4a70951640 1 -- 172.21.15.71:0/1591292959 shutdown_connections 2024-03-29T11:05:44.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.492+0000 7f4a70951640 1 -- 172.21.15.71:0/1591292959 wait complete. 2024-03-29T11:05:44.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:44.493+0000 7f4a70951640 1 librados: shutdown 2024-03-29T11:05:44.493 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:05:51.300 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:05:51.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.478+0000 7fb75ce99640 10 monclient: get_monmap_and_config 2024-03-29T11:05:51.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.478+0000 7fb75ce99640 10 monclient: build_initial_monmap 2024-03-29T11:05:51.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.478+0000 7fb75ce99640 10 monclient: monmap: 2024-03-29T11:05:51.460 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:05:51.460 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:51.460 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:05:51.479371+0000 2024-03-29T11:05:51.460 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:05:51.479371+0000 2024-03-29T11:05:51.460 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:51.460 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:51.461 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:05:51.461 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:05:51.461 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:05:51.461 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:51.462 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.482+0000 7fb75ce99640 1 Processor -- start 2024-03-29T11:05:51.462 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.482+0000 7fb75ce99640 1 -- start start 2024-03-29T11:05:51.462 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.482+0000 7fb75ce99640 10 monclient: init 2024-03-29T11:05:51.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.483+0000 7fb75ce99640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:51.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.483+0000 7fb75ce99640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:05:51.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.484+0000 7fb75ce99640 10 monclient(hunting): picked mon.noname-b con 0x7fb750006a70 addr v1:172.21.15.71:6790/0 2024-03-29T11:05:51.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.484+0000 7fb75ce99640 10 monclient(hunting): picked mon.noname-c con 0x7fb75002b100 addr v1:172.21.15.137:6789/0 2024-03-29T11:05:51.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.484+0000 7fb75ce99640 10 monclient(hunting): picked mon.noname-a con 0x7fb750029c30 addr v1:172.21.15.71:6789/0 2024-03-29T11:05:51.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.484+0000 7fb75ce99640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb75002a270 con 0x7fb750029c30 2024-03-29T11:05:51.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.484+0000 7fb75ce99640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb75002c440 con 0x7fb75002b100 2024-03-29T11:05:51.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.484+0000 7fb75ce99640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7500c9440 con 0x7fb750006a70 2024-03-29T11:05:51.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.484+0000 7fb75ce99640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:51.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.484+0000 7fb75ce99640 10 monclient(hunting): authenticate will time out at 1107.742206s 2024-03-29T11:05:51.467 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.488+0000 7fb756ffd640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fb75002b100 0x7fb7500cf100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:49397/0 (socket says 172.21.15.71:49397) 2024-03-29T11:05:51.467 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.488+0000 7fb756ffd640 1 -- 172.21.15.71:0/636860684 learned_addr learned my addr 172.21.15.71:0/636860684 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:05:51.469 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.490+0000 7fb7567fc640 1 -- 172.21.15.71:0/636860684 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1933273974 0 0) 0x7fb75002c440 con 0x7fb75002b100 2024-03-29T11:05:51.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.493+0000 7fb7567fc640 10 monclient(hunting): _negotiate 2024-03-29T11:05:51.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.493+0000 7fb7567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:51.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.493+0000 7fb7567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:51.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.493+0000 7fb7567fc640 10 monclient(hunting): my global_id is 4396 2024-03-29T11:05:51.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.493+0000 7fb7567fc640 1 -- 172.21.15.71:0/636860684 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb740003520 con 0x7fb75002b100 2024-03-29T11:05:51.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.493+0000 7fb7567fc640 1 -- 172.21.15.71:0/636860684 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3579520947 0 0) 0x7fb75002a270 con 0x7fb750029c30 2024-03-29T11:05:51.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.493+0000 7fb7567fc640 10 monclient(hunting): _negotiate 2024-03-29T11:05:51.474 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.493+0000 7fb7567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:51.474 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.493+0000 7fb7567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:51.474 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.493+0000 7fb7567fc640 10 monclient(hunting): my global_id is 4428 2024-03-29T11:05:51.474 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.493+0000 7fb7567fc640 1 -- 172.21.15.71:0/636860684 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb75002c440 con 0x7fb750029c30 2024-03-29T11:05:51.474 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.493+0000 7fb7567fc640 1 -- 172.21.15.71:0/636860684 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356176085 0 0) 0x7fb7500c9440 con 0x7fb750006a70 2024-03-29T11:05:51.474 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.493+0000 7fb7567fc640 10 monclient(hunting): _negotiate 2024-03-29T11:05:51.474 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.493+0000 7fb7567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:51.474 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.493+0000 7fb7567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:51.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.493+0000 7fb7567fc640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:05:51.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.493+0000 7fb7567fc640 1 -- 172.21.15.71:0/636860684 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb75002a270 con 0x7fb750006a70 2024-03-29T11:05:51.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.494+0000 7fb7567fc640 1 -- 172.21.15.71:0/636860684 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 789157330 0 0) 0x7fb740003520 con 0x7fb75002b100 2024-03-29T11:05:51.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.494+0000 7fb7567fc640 1 -- 172.21.15.71:0/636860684 >> v1:172.21.15.71:6790/0 conn(0x7fb750006a70 legacy=0x7fb7500c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:51.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.494+0000 7fb7567fc640 1 -- 172.21.15.71:0/636860684 >> v1:172.21.15.71:6789/0 conn(0x7fb750029c30 legacy=0x7fb7500d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:51.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.494+0000 7fb7567fc640 10 monclient: _finish_hunting 0 2024-03-29T11:05:51.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.494+0000 7fb7567fc640 1 monclient: found mon.noname-c 2024-03-29T11:05:51.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.494+0000 7fb7567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:51.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.494+0000 7fb7567fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.137:6789/0 2024-03-29T11:05:51.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.494+0000 7fb7567fc640 1 -- 172.21.15.71:0/636860684 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7500c95d0 con 0x7fb75002b100 2024-03-29T11:05:51.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.494+0000 7fb7567fc640 10 monclient: _finish_auth 0 2024-03-29T11:05:51.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.494+0000 7fb7567fc640 10 monclient: _check_auth_tickets 2024-03-29T11:05:51.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.494+0000 7fb7567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:51.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.494+0000 7fb7567fc640 1 -- 172.21.15.71:0/636860684 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb74c001690 con 0x7fb75002b100 2024-03-29T11:05:51.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.494+0000 7fb7567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:51.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.494+0000 7fb7567fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:05:51.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.494+0000 7fb7567fc640 10 monclient: dump: 2024-03-29T11:05:51.477 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:51.477 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:51.477 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:51.477 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:51.477 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:51.477 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:51.477 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:51.477 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:51.477 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:51.477 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:51.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.494+0000 7fb75ce99640 5 monclient: authenticate success, global_id 4396 2024-03-29T11:05:51.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.494+0000 7fb75ce99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:05:51.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.494+0000 7fb7567fc640 1 -- 172.21.15.71:0/636860684 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb74c0017f0 con 0x7fb75002b100 2024-03-29T11:05:51.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.494+0000 7fb7567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:51.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.494+0000 7fb7567fc640 1 -- 172.21.15.71:0/636860684 <== mon.1 v1:172.21.15.137:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb74c001c00 con 0x7fb75002b100 2024-03-29T11:05:51.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.494+0000 7fb7567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:51.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.494+0000 7fb7567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:05:51.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.494+0000 7fb7567fc640 10 monclient: dump: 2024-03-29T11:05:51.478 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:51.478 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:51.478 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:51.478 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:51.478 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:51.478 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:51.478 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:51.478 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:51.478 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:51.478 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:51.479 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.494+0000 7fb75ce99640 10 monclient: get_monmap_and_config success 2024-03-29T11:05:51.479 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.494+0000 7fb75ce99640 4 set_mon_vals no callback set 2024-03-29T11:05:51.479 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.495+0000 7fb75ce99640 10 monclient: shutdown 2024-03-29T11:05:51.479 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.495+0000 7fb75ce99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:51.479 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.495+0000 7fb75ce99640 1 -- 172.21.15.71:0/636860684 >> v1:172.21.15.137:6789/0 conn(0x7fb75002b100 legacy=0x7fb7500cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:51.479 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.496+0000 7fb75ce99640 1 -- 172.21.15.71:0/636860684 shutdown_connections 2024-03-29T11:05:51.479 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.496+0000 7fb75ce99640 1 -- 172.21.15.71:0/636860684 >> 172.21.15.71:0/636860684 conn(0x7fb7500058f0 msgr2=0x7fb750005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:51.479 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.496+0000 7fb75ce99640 1 -- 172.21.15.71:0/636860684 shutdown_connections 2024-03-29T11:05:51.480 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.496+0000 7fb75ce99640 1 -- 172.21.15.71:0/636860684 wait complete. 2024-03-29T11:05:51.480 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.497+0000 7fb75ce99640 10 monclient: build_initial_monmap 2024-03-29T11:05:51.480 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.497+0000 7fb75ce99640 10 monclient: monmap: 2024-03-29T11:05:51.480 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:05:51.480 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:05:51.480 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:05:51.480 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:05:51.480 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:51.480 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:51.480 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:05:51.481 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:05:51.481 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:05:51.481 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:51.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.497+0000 7fb75ce99640 1 librados: starting msgr at 2024-03-29T11:05:51.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.497+0000 7fb75ce99640 1 librados: starting objecter 2024-03-29T11:05:51.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.498+0000 7fb75ce99640 1 Processor -- start 2024-03-29T11:05:51.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.498+0000 7fb75ce99640 1 -- start start 2024-03-29T11:05:51.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.498+0000 7fb75ce99640 1 librados: setting wanted keys 2024-03-29T11:05:51.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.498+0000 7fb75ce99640 1 librados: calling monclient init 2024-03-29T11:05:51.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.498+0000 7fb75ce99640 10 monclient: init 2024-03-29T11:05:51.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.499+0000 7fb75ce99640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:51.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.499+0000 7fb75ce99640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:05:51.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.499+0000 7fb75ce99640 10 monclient(hunting): picked mon.noname-b con 0x7fb750006a70 addr v1:172.21.15.137:6789/0 2024-03-29T11:05:51.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.499+0000 7fb75ce99640 10 monclient(hunting): picked mon.noname-a con 0x7fb750029c30 addr v1:172.21.15.71:6789/0 2024-03-29T11:05:51.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.499+0000 7fb75ce99640 10 monclient(hunting): picked mon.noname-c con 0x7fb750164d70 addr v1:172.21.15.71:6790/0 2024-03-29T11:05:51.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.499+0000 7fb75ce99640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7501652e0 con 0x7fb750029c30 2024-03-29T11:05:51.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.499+0000 7fb75ce99640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb750032130 con 0x7fb750006a70 2024-03-29T11:05:51.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.499+0000 7fb75ce99640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7500322c0 con 0x7fb750164d70 2024-03-29T11:05:51.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.499+0000 7fb75ce99640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:51.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.499+0000 7fb75ce99640 10 monclient(hunting): authenticate will time out at 1107.757565s 2024-03-29T11:05:51.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.503+0000 7fb757fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fb750164d70 0x7fb75002de50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:47297/0 (socket says 172.21.15.71:47297) 2024-03-29T11:05:51.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.503+0000 7fb757fff640 1 -- 172.21.15.71:0/1704108402 learned_addr learned my addr 172.21.15.71:0/1704108402 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:05:51.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.506+0000 7fb73ffff640 10 client.?.objecter ms_handle_connect 0x7fb750164d70 2024-03-29T11:05:51.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.506+0000 7fb73ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:51.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.506+0000 7fb73ffff640 1 -- 172.21.15.71:0/1704108402 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1135708904 0 0) 0x7fb7500322c0 con 0x7fb750164d70 2024-03-29T11:05:51.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.506+0000 7fb73ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:05:51.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.506+0000 7fb73ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:51.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.506+0000 7fb73ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:51.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.506+0000 7fb73ffff640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:05:51.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.506+0000 7fb73ffff640 1 -- 172.21.15.71:0/1704108402 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb744004eb0 con 0x7fb750164d70 2024-03-29T11:05:51.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.507+0000 7fb73ffff640 10 client.?.objecter ms_handle_connect 0x7fb750006a70 2024-03-29T11:05:51.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.508+0000 7fb73ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:51.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.508+0000 7fb73ffff640 1 -- 172.21.15.71:0/1704108402 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739890167 0 0) 0x7fb750032130 con 0x7fb750006a70 2024-03-29T11:05:51.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.508+0000 7fb73ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:05:51.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.514+0000 7fb73ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:51.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.514+0000 7fb73ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:51.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.514+0000 7fb73ffff640 10 monclient(hunting): my global_id is 4399 2024-03-29T11:05:51.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.514+0000 7fb73ffff640 1 -- 172.21.15.71:0/1704108402 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7500322c0 con 0x7fb750006a70 2024-03-29T11:05:51.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.514+0000 7fb73ffff640 1 -- 172.21.15.71:0/1704108402 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3709753516 0 0) 0x7fb744004eb0 con 0x7fb750164d70 2024-03-29T11:05:51.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.514+0000 7fb73ffff640 1 -- 172.21.15.71:0/1704108402 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb750032130 con 0x7fb750164d70 2024-03-29T11:05:51.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.514+0000 7fb73ffff640 1 -- 172.21.15.71:0/1704108402 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb75804fdd0 con 0x7fb750164d70 2024-03-29T11:05:51.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.514+0000 7fb73ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:51.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.514+0000 7fb73ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:05:51.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.514+0000 7fb73ffff640 10 monclient(hunting): dump: 2024-03-29T11:05:51.495 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:51.495 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:51.495 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:51.495 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:51.495 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:51.495 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:51.495 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:51.495 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:51.495 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:51.495 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:51.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.514+0000 7fb73ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:05:51.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.515+0000 7fb73ffff640 10 client.?.objecter ms_handle_connect 0x7fb750029c30 2024-03-29T11:05:51.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.515+0000 7fb73ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:51.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.515+0000 7fb73ffff640 1 -- 172.21.15.71:0/1704108402 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3065407974 0 0) 0x7fb750032130 con 0x7fb750164d70 2024-03-29T11:05:51.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.515+0000 7fb73ffff640 1 -- 172.21.15.71:0/1704108402 >> v1:172.21.15.137:6789/0 conn(0x7fb750006a70 legacy=0x7fb750164660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:51.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.515+0000 7fb73ffff640 1 -- 172.21.15.71:0/1704108402 >> v1:172.21.15.71:6789/0 conn(0x7fb750029c30 legacy=0x7fb75002d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:51.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.516+0000 7fb73ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:05:51.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.516+0000 7fb73ffff640 1 monclient: found mon.c 2024-03-29T11:05:51.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.516+0000 7fb73ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:51.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.516+0000 7fb73ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:05:51.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.516+0000 7fb73ffff640 1 -- 172.21.15.71:0/1704108402 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb750032450 con 0x7fb750164d70 2024-03-29T11:05:51.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.516+0000 7fb73ffff640 10 monclient: _finish_auth 0 2024-03-29T11:05:51.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.516+0000 7fb73ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:05:51.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.516+0000 7fb73ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:51.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.517+0000 7fb75ce99640 5 monclient: authenticate success, global_id 4454 2024-03-29T11:05:51.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.517+0000 7fb75ce99640 10 monclient: _renew_subs 2024-03-29T11:05:51.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.517+0000 7fb75ce99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:05:51.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.517+0000 7fb75ce99640 1 -- 172.21.15.71:0/1704108402 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb75016c9b0 con 0x7fb750164d70 2024-03-29T11:05:51.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.517+0000 7fb75ce99640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:05:51.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.517+0000 7fb75ce99640 10 monclient: _renew_subs 2024-03-29T11:05:51.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.517+0000 7fb75ce99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:05:51.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.517+0000 7fb75ce99640 1 -- 172.21.15.71:0/1704108402 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb75016cd70 con 0x7fb750164d70 2024-03-29T11:05:51.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.517+0000 7fb75ce99640 1 librados: init done 2024-03-29T11:05:51.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.521+0000 7fb73ffff640 1 -- 172.21.15.71:0/1704108402 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb75804f400 con 0x7fb750164d70 2024-03-29T11:05:51.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.521+0000 7fb73ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:51.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.521+0000 7fb73ffff640 1 -- 172.21.15.71:0/1704108402 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb758075f10 con 0x7fb750164d70 2024-03-29T11:05:51.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.521+0000 7fb73ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:51.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.521+0000 7fb73ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:05:51.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.521+0000 7fb73ffff640 10 monclient: dump: 2024-03-29T11:05:51.501 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:51.501 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:51.501 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:51.501 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:51.501 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:51.501 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:51.501 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:51.501 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:51.501 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:51.501 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:51.502 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.522+0000 7fb7557fa640 4 set_mon_vals no callback set 2024-03-29T11:05:51.502 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.523+0000 7fb75ce99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:51.502 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.523+0000 7fb73ffff640 1 -- 172.21.15.71:0/1704108402 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7fb758067bf0 con 0x7fb750164d70 2024-03-29T11:05:51.502 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.523+0000 7fb75ce99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:51.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.523+0000 7fb75ce99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:05:51.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.523+0000 7fb75ce99640 1 -- 172.21.15.71:0/1704108402 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb738005080 con 0x7fb750164d70 2024-03-29T11:05:51.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.524+0000 7fb73ffff640 1 -- 172.21.15.71:0/1704108402 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7fb7580ff550 con 0x7fb750164d70 2024-03-29T11:05:51.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.524+0000 7fb73ffff640 10 client.4454.objecter ms_dispatch 0x7fb758000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:05:51.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.524+0000 7fb73ffff640 3 client.4454.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:05:51.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.524+0000 7fb73ffff640 3 client.4454.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:05:51.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.524+0000 7fb73ffff640 20 client.4454.objecter dump_active .. 0 homeless 2024-03-29T11:05:51.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.524+0000 7fb7577fe640 10 monclient: get_auth_request con 0x7fb74406dbb0 auth_method 0 2024-03-29T11:05:51.505 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.526+0000 7fb73ffff640 10 client.4454.objecter ms_handle_connect 0x7fb74406dbb0 2024-03-29T11:05:51.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.535+0000 7fb73ffff640 1 -- 172.21.15.71:0/1704108402 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb7580cae30 con 0x7fb750164d70 2024-03-29T11:05:51.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.535+0000 7fb73ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:51.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.535+0000 7fb73ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:05:51.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.895+0000 7fb75ce99640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:51.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.895+0000 7fb75ce99640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:51.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.895+0000 7fb75ce99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:05:51.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.895+0000 7fb75ce99640 1 -- 172.21.15.71:0/1704108402 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb738005330 con 0x7fb750164d70 2024-03-29T11:05:51.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.896+0000 7fb73ffff640 1 -- 172.21.15.71:0/1704108402 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7fb7580ccd60 con 0x7fb750164d70 2024-03-29T11:05:51.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.896+0000 7fb73ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:51.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.896+0000 7fb73ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:05:51.876 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:05:51.876 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:05:51.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.899+0000 7fb75ce99640 10 librados: watch_flush enter 2024-03-29T11:05:51.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.899+0000 7fb75ce99640 10 librados: watch_flush exit 2024-03-29T11:05:51.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.899+0000 7fb75ce99640 20 client.4454.objecter shutdown clearing up homeless session... 2024-03-29T11:05:51.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.899+0000 7fb75ce99640 10 client.4454.objecter successfully canceled tick 2024-03-29T11:05:51.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.899+0000 7fb75ce99640 1 -- 172.21.15.71:0/1704108402 >> v1:172.21.15.137:6816/973332665 conn(0x7fb74406dbb0 legacy=0x7fb744070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:51.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.899+0000 7fb75ce99640 10 monclient: shutdown 2024-03-29T11:05:51.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.899+0000 7fb75ce99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:51.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.899+0000 7fb75ce99640 1 -- 172.21.15.71:0/1704108402 >> v1:172.21.15.71:6790/0 conn(0x7fb750164d70 legacy=0x7fb75002de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:51.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.900+0000 7fb75ce99640 1 -- 172.21.15.71:0/1704108402 shutdown_connections 2024-03-29T11:05:51.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.900+0000 7fb75ce99640 1 -- 172.21.15.71:0/1704108402 >> 172.21.15.71:0/1704108402 conn(0x7fb7500058f0 msgr2=0x7fb7500c6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:51.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.900+0000 7fb75ce99640 1 -- 172.21.15.71:0/1704108402 shutdown_connections 2024-03-29T11:05:51.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.900+0000 7fb75ce99640 1 -- 172.21.15.71:0/1704108402 wait complete. 2024-03-29T11:05:51.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:51.900+0000 7fb75ce99640 1 librados: shutdown 2024-03-29T11:05:51.894 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:05:58.701 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:05:58.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.877+0000 7f98d552b640 10 monclient: get_monmap_and_config 2024-03-29T11:05:58.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.877+0000 7f98d552b640 10 monclient: build_initial_monmap 2024-03-29T11:05:58.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.877+0000 7f98d552b640 10 monclient: monmap: 2024-03-29T11:05:58.860 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:05:58.860 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:58.861 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:05:58.878354+0000 2024-03-29T11:05:58.861 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:05:58.878354+0000 2024-03-29T11:05:58.861 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:58.861 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:58.861 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:05:58.861 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:05:58.861 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:05:58.861 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:58.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.878+0000 7f98d552b640 1 Processor -- start 2024-03-29T11:05:58.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.878+0000 7f98d552b640 1 -- start start 2024-03-29T11:05:58.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.878+0000 7f98d552b640 10 monclient: init 2024-03-29T11:05:58.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.879+0000 7f98d552b640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:58.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.879+0000 7f98d552b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:05:58.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.879+0000 7f98d552b640 10 monclient(hunting): picked mon.noname-a con 0x7f98d00b4820 addr v1:172.21.15.71:6789/0 2024-03-29T11:05:58.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.880+0000 7f98d552b640 10 monclient(hunting): picked mon.noname-b con 0x7f98d00a9ce0 addr v1:172.21.15.71:6790/0 2024-03-29T11:05:58.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.880+0000 7f98d552b640 10 monclient(hunting): picked mon.noname-c con 0x7f98d00b3cd0 addr v1:172.21.15.137:6789/0 2024-03-29T11:05:58.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.880+0000 7f98d552b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d00b4180 con 0x7f98d00b4820 2024-03-29T11:05:58.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.880+0000 7f98d552b640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d00aaf60 con 0x7f98d00b3cd0 2024-03-29T11:05:58.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.880+0000 7f98d552b640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d013c200 con 0x7f98d00a9ce0 2024-03-29T11:05:58.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.880+0000 7f98d552b640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:58.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.880+0000 7f98d552b640 10 monclient(hunting): authenticate will time out at 1115.138659s 2024-03-29T11:05:58.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.880+0000 7f98cf7fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f98d00a9ce0 0x7f98d0145770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:34434/0 (socket says 172.21.15.71:34434) 2024-03-29T11:05:58.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.880+0000 7f98cf7fe640 1 -- 172.21.15.71:0/3888493936 learned_addr learned my addr 172.21.15.71:0/3888493936 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:05:58.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.883+0000 7f98ceffd640 1 -- 172.21.15.71:0/3888493936 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735420045 0 0) 0x7f98d013c200 con 0x7f98d00a9ce0 2024-03-29T11:05:58.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.883+0000 7f98ceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:05:58.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.883+0000 7f98ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:58.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.883+0000 7f98ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:58.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.883+0000 7f98ceffd640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:05:58.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.884+0000 7f98ceffd640 1 -- 172.21.15.71:0/3888493936 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98bc003520 con 0x7f98d00a9ce0 2024-03-29T11:05:58.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.884+0000 7f98ceffd640 1 -- 172.21.15.71:0/3888493936 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443453693 0 0) 0x7f98d00aaf60 con 0x7f98d00b3cd0 2024-03-29T11:05:58.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.884+0000 7f98ceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:05:58.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.884+0000 7f98ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:58.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.884+0000 7f98ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:58.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.884+0000 7f98ceffd640 10 monclient(hunting): my global_id is 4402 2024-03-29T11:05:58.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.884+0000 7f98ceffd640 1 -- 172.21.15.71:0/3888493936 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98d013c200 con 0x7f98d00b3cd0 2024-03-29T11:05:58.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.884+0000 7f98ceffd640 1 -- 172.21.15.71:0/3888493936 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 168379725 0 0) 0x7f98d013c200 con 0x7f98d00b3cd0 2024-03-29T11:05:58.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.884+0000 7f98ceffd640 1 -- 172.21.15.71:0/3888493936 >> v1:172.21.15.71:6790/0 conn(0x7f98d00a9ce0 legacy=0x7f98d0145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:58.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.884+0000 7f98ceffd640 1 -- 172.21.15.71:0/3888493936 >> v1:172.21.15.71:6789/0 conn(0x7f98d00b4820 legacy=0x7f98d013baf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:05:58.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.885+0000 7f98ceffd640 10 monclient: _finish_hunting 0 2024-03-29T11:05:58.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.885+0000 7f98ceffd640 1 monclient: found mon.noname-c 2024-03-29T11:05:58.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.885+0000 7f98ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:58.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.885+0000 7f98ceffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.137:6789/0 2024-03-29T11:05:58.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.885+0000 7f98ceffd640 1 -- 172.21.15.71:0/3888493936 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d0145e80 con 0x7f98d00b3cd0 2024-03-29T11:05:58.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.885+0000 7f98ceffd640 10 monclient: _finish_auth 0 2024-03-29T11:05:58.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.885+0000 7f98ceffd640 10 monclient: _check_auth_tickets 2024-03-29T11:05:58.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.885+0000 7f98ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:58.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.885+0000 7f98ceffd640 1 -- 172.21.15.71:0/3888493936 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f98c80019f0 con 0x7f98d00b3cd0 2024-03-29T11:05:58.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.885+0000 7f98ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:58.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.885+0000 7f98ceffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:05:58.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.885+0000 7f98ceffd640 10 monclient: dump: 2024-03-29T11:05:58.869 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:58.869 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:58.869 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:58.869 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:58.869 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:58.869 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:58.869 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:58.869 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:58.869 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:58.870 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:58.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.885+0000 7f98d552b640 5 monclient: authenticate success, global_id 4402 2024-03-29T11:05:58.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.885+0000 7f98d552b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:05:58.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.887+0000 7f98ceffd640 1 -- 172.21.15.71:0/3888493936 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98c8001b50 con 0x7f98d00b3cd0 2024-03-29T11:05:58.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.887+0000 7f98ceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:58.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.887+0000 7f98ceffd640 1 -- 172.21.15.71:0/3888493936 <== mon.1 v1:172.21.15.137:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f98c8001fa0 con 0x7f98d00b3cd0 2024-03-29T11:05:58.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.887+0000 7f98d552b640 10 monclient: get_monmap_and_config success 2024-03-29T11:05:58.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.887+0000 7f98d552b640 4 set_mon_vals no callback set 2024-03-29T11:05:58.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.887+0000 7f98ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:58.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.888+0000 7f98ceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:05:58.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.888+0000 7f98ceffd640 10 monclient: dump: 2024-03-29T11:05:58.870 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:58.871 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:58.871 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:58.871 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:58.871 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:58.871 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:58.871 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:58.871 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:58.871 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:58.871 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:58.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.888+0000 7f98d552b640 10 monclient: shutdown 2024-03-29T11:05:58.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.888+0000 7f98d552b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:58.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.888+0000 7f98d552b640 1 -- 172.21.15.71:0/3888493936 >> v1:172.21.15.137:6789/0 conn(0x7f98d00b3cd0 legacy=0x7f98d014d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:58.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.888+0000 7f98d552b640 1 -- 172.21.15.71:0/3888493936 shutdown_connections 2024-03-29T11:05:58.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.888+0000 7f98d552b640 1 -- 172.21.15.71:0/3888493936 >> 172.21.15.71:0/3888493936 conn(0x7f98d006d070 msgr2=0x7f98d0066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:58.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.889+0000 7f98d552b640 1 -- 172.21.15.71:0/3888493936 shutdown_connections 2024-03-29T11:05:58.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.889+0000 7f98d552b640 1 -- 172.21.15.71:0/3888493936 wait complete. 2024-03-29T11:05:58.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.890+0000 7f98d552b640 10 monclient: build_initial_monmap 2024-03-29T11:05:58.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.890+0000 7f98d552b640 10 monclient: monmap: 2024-03-29T11:05:58.873 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:05:58.873 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:05:58.873 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:05:58.873 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:05:58.873 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:58.873 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:58.873 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:05:58.873 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:05:58.873 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:05:58.873 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:58.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.890+0000 7f98d552b640 1 librados: starting msgr at 2024-03-29T11:05:58.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.890+0000 7f98d552b640 1 librados: starting objecter 2024-03-29T11:05:58.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.891+0000 7f98d552b640 1 Processor -- start 2024-03-29T11:05:58.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.891+0000 7f98d552b640 1 -- start start 2024-03-29T11:05:58.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.891+0000 7f98d552b640 1 librados: setting wanted keys 2024-03-29T11:05:58.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.891+0000 7f98d552b640 1 librados: calling monclient init 2024-03-29T11:05:58.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.891+0000 7f98d552b640 10 monclient: init 2024-03-29T11:05:58.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.892+0000 7f98d552b640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:58.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.892+0000 7f98d552b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:05:58.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.892+0000 7f98d552b640 10 monclient(hunting): picked mon.noname-c con 0x7f98d00a9ce0 addr v1:172.21.15.71:6790/0 2024-03-29T11:05:58.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.895+0000 7f98d552b640 10 monclient(hunting): picked mon.noname-a con 0x7f98d00b3cd0 addr v1:172.21.15.71:6789/0 2024-03-29T11:05:58.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.895+0000 7f98d552b640 10 monclient(hunting): picked mon.noname-b con 0x7f98d00b4820 addr v1:172.21.15.137:6789/0 2024-03-29T11:05:58.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.895+0000 7f98d552b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d013b370 con 0x7f98d00b3cd0 2024-03-29T11:05:58.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.895+0000 7f98d552b640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d013b5e0 con 0x7f98d00b4820 2024-03-29T11:05:58.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.895+0000 7f98d552b640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d013f300 con 0x7f98d00a9ce0 2024-03-29T11:05:58.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.895+0000 7f98d552b640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:58.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.895+0000 7f98d552b640 10 monclient(hunting): authenticate will time out at 1115.153521s 2024-03-29T11:05:58.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.895+0000 7f98cffff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f98d00a9ce0 0x7f98d0136690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:35908/0 (socket says 172.21.15.71:35908) 2024-03-29T11:05:58.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.896+0000 7f98cffff640 1 -- 172.21.15.71:0/2213267697 learned_addr learned my addr 172.21.15.71:0/2213267697 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:05:58.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.896+0000 7f98ccff9640 10 client.?.objecter ms_handle_connect 0x7f98d00a9ce0 2024-03-29T11:05:58.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.896+0000 7f98ccff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:58.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.897+0000 7f98ccff9640 1 -- 172.21.15.71:0/2213267697 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975593069 0 0) 0x7f98d013f300 con 0x7f98d00a9ce0 2024-03-29T11:05:58.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.897+0000 7f98ccff9640 10 monclient(hunting): _negotiate 2024-03-29T11:05:58.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.897+0000 7f98ccff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:58.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.897+0000 7f98ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:58.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.897+0000 7f98ccff9640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:05:58.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.897+0000 7f98ccff9640 1 -- 172.21.15.71:0/2213267697 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98b0004eb0 con 0x7f98d00a9ce0 2024-03-29T11:05:58.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.897+0000 7f98ccff9640 10 client.?.objecter ms_handle_connect 0x7f98d00b3cd0 2024-03-29T11:05:58.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.897+0000 7f98ccff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:58.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.898+0000 7f98ccff9640 1 -- 172.21.15.71:0/2213267697 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3933829371 0 0) 0x7f98d013b370 con 0x7f98d00b3cd0 2024-03-29T11:05:58.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.898+0000 7f98ccff9640 10 monclient(hunting): _negotiate 2024-03-29T11:05:58.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.898+0000 7f98ccff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:58.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.898+0000 7f98ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:58.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.898+0000 7f98ccff9640 10 monclient(hunting): my global_id is 4434 2024-03-29T11:05:58.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.898+0000 7f98ccff9640 1 -- 172.21.15.71:0/2213267697 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98d013f300 con 0x7f98d00b3cd0 2024-03-29T11:05:58.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.898+0000 7f98ccff9640 1 -- 172.21.15.71:0/2213267697 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3004025705 0 0) 0x7f98b0004eb0 con 0x7f98d00a9ce0 2024-03-29T11:05:58.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.898+0000 7f98ccff9640 1 -- 172.21.15.71:0/2213267697 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98d013b370 con 0x7f98d00a9ce0 2024-03-29T11:05:58.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.898+0000 7f98ccff9640 1 -- 172.21.15.71:0/2213267697 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f98b8001750 con 0x7f98d00a9ce0 2024-03-29T11:05:58.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.898+0000 7f98ccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:58.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.898+0000 7f98ccff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:05:58.879 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.898+0000 7f98ccff9640 10 monclient(hunting): dump: 2024-03-29T11:05:58.879 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:58.879 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:58.879 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:58.880 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:58.880 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:58.880 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:58.880 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:58.880 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:58.880 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:58.880 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:58.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.898+0000 7f98ccff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:05:58.880 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.899+0000 7f98ccff9640 10 client.?.objecter ms_handle_connect 0x7f98d00b4820 2024-03-29T11:05:58.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.902+0000 7f98ccff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:58.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.902+0000 7f98ccff9640 1 -- 172.21.15.71:0/2213267697 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 885525741 0 0) 0x7f98d013f300 con 0x7f98d00b3cd0 2024-03-29T11:05:58.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.902+0000 7f98ccff9640 1 -- 172.21.15.71:0/2213267697 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98b0004eb0 con 0x7f98d00b3cd0 2024-03-29T11:05:58.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.902+0000 7f98ccff9640 1 -- 172.21.15.71:0/2213267697 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 606108800 0 0) 0x7f98d013b370 con 0x7f98d00a9ce0 2024-03-29T11:05:58.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.902+0000 7f98ccff9640 1 -- 172.21.15.71:0/2213267697 >> v1:172.21.15.137:6789/0 conn(0x7f98d00b4820 legacy=0x7f98d0132d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:58.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.902+0000 7f98ccff9640 1 -- 172.21.15.71:0/2213267697 >> v1:172.21.15.71:6789/0 conn(0x7f98d00b3cd0 legacy=0x7f98d01da2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:58.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.902+0000 7f98ccff9640 10 monclient: _finish_hunting 0 2024-03-29T11:05:58.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.902+0000 7f98ccff9640 1 monclient: found mon.c 2024-03-29T11:05:58.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.902+0000 7f98ccff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:58.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.902+0000 7f98ccff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:05:58.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.902+0000 7f98ccff9640 1 -- 172.21.15.71:0/2213267697 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d013f490 con 0x7f98d00a9ce0 2024-03-29T11:05:58.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.902+0000 7f98ccff9640 10 monclient: _finish_auth 0 2024-03-29T11:05:58.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.902+0000 7f98ccff9640 10 monclient: _check_auth_tickets 2024-03-29T11:05:58.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.902+0000 7f98ccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:58.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.903+0000 7f98d552b640 5 monclient: authenticate success, global_id 4460 2024-03-29T11:05:58.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.903+0000 7f98d552b640 10 monclient: _renew_subs 2024-03-29T11:05:58.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.903+0000 7f98d552b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:05:58.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.903+0000 7f98d552b640 1 -- 172.21.15.71:0/2213267697 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d013a720 con 0x7f98d00a9ce0 2024-03-29T11:05:58.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.903+0000 7f98d552b640 10 client.4460.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:05:58.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.903+0000 7f98d552b640 10 monclient: _renew_subs 2024-03-29T11:05:58.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.903+0000 7f98d552b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:05:58.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.903+0000 7f98d552b640 1 -- 172.21.15.71:0/2213267697 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d0136190 con 0x7f98d00a9ce0 2024-03-29T11:05:58.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.903+0000 7f98d552b640 1 librados: init done 2024-03-29T11:05:58.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.903+0000 7f98ccff9640 1 -- 172.21.15.71:0/2213267697 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98b8001c60 con 0x7f98d00a9ce0 2024-03-29T11:05:58.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.903+0000 7f98ccff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:58.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.903+0000 7f98ccff9640 1 -- 172.21.15.71:0/2213267697 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f98b8004a00 con 0x7f98d00a9ce0 2024-03-29T11:05:58.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.903+0000 7f98ccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:58.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.903+0000 7f98ccff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:05:58.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.903+0000 7f98ccff9640 10 monclient: dump: 2024-03-29T11:05:58.885 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:05:58.885 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:05:58.885 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:58.885 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:05:58.885 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:58.885 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:05:58.885 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:05:58.885 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:05:58.885 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:05:58.885 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:05:58.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.903+0000 7f98cdffb640 4 set_mon_vals no callback set 2024-03-29T11:05:58.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.904+0000 7f98d552b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:58.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.905+0000 7f98d552b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:58.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.905+0000 7f98d552b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:05:58.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.905+0000 7f98d552b640 1 -- 172.21.15.71:0/2213267697 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b4005080 con 0x7f98d00a9ce0 2024-03-29T11:05:58.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.907+0000 7f98ccff9640 1 -- 172.21.15.71:0/2213267697 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f98b801a9d0 con 0x7f98d00a9ce0 2024-03-29T11:05:58.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.908+0000 7f98cf7fe640 10 monclient: get_auth_request con 0x7f98b006dd80 auth_method 0 2024-03-29T11:05:58.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.908+0000 7f98ccff9640 1 -- 172.21.15.71:0/2213267697 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f98b8084940 con 0x7f98d00a9ce0 2024-03-29T11:05:58.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.908+0000 7f98ccff9640 10 client.4460.objecter ms_dispatch 0x7f98d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:05:58.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.909+0000 7f98ccff9640 3 client.4460.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:05:58.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.909+0000 7f98ccff9640 3 client.4460.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:05:58.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.909+0000 7f98ccff9640 20 client.4460.objecter dump_active .. 0 homeless 2024-03-29T11:05:58.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.909+0000 7f98ccff9640 10 client.4460.objecter ms_handle_connect 0x7f98b006dd80 2024-03-29T11:05:58.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.920+0000 7f98ccff9640 1 -- 172.21.15.71:0/2213267697 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f98b8050220 con 0x7f98d00a9ce0 2024-03-29T11:05:58.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.920+0000 7f98ccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:58.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:58.920+0000 7f98ccff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:05:59.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:59.279+0000 7f98d552b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:59.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:59.279+0000 7f98d552b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:59.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:59.279+0000 7f98d552b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:05:59.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:59.279+0000 7f98d552b640 1 -- 172.21.15.71:0/2213267697 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98b4005330 con 0x7f98d00a9ce0 2024-03-29T11:05:59.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:59.282+0000 7f98ccff9640 1 -- 172.21.15.71:0/2213267697 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f98b8052150 con 0x7f98d00a9ce0 2024-03-29T11:05:59.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:59.282+0000 7f98ccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:59.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:59.282+0000 7f98ccff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:05:59.261 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:05:59.261 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:05:59.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:59.286+0000 7f98d552b640 10 librados: watch_flush enter 2024-03-29T11:05:59.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:59.286+0000 7f98d552b640 10 librados: watch_flush exit 2024-03-29T11:05:59.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:59.286+0000 7f98d552b640 20 client.4460.objecter shutdown clearing up homeless session... 2024-03-29T11:05:59.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:59.286+0000 7f98d552b640 10 client.4460.objecter successfully canceled tick 2024-03-29T11:05:59.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:59.286+0000 7f98d552b640 1 -- 172.21.15.71:0/2213267697 >> v1:172.21.15.137:6816/973332665 conn(0x7f98b006dd80 legacy=0x7f98b0070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:59.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:59.286+0000 7f98d552b640 10 monclient: shutdown 2024-03-29T11:05:59.266 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:59.286+0000 7f98d552b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:59.266 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:59.286+0000 7f98d552b640 1 -- 172.21.15.71:0/2213267697 >> v1:172.21.15.71:6790/0 conn(0x7f98d00a9ce0 legacy=0x7f98d0136690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:59.266 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:59.287+0000 7f98d552b640 1 -- 172.21.15.71:0/2213267697 shutdown_connections 2024-03-29T11:05:59.266 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:59.287+0000 7f98d552b640 1 -- 172.21.15.71:0/2213267697 >> 172.21.15.71:0/2213267697 conn(0x7f98d006d070 msgr2=0x7f98d00b8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:59.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:59.288+0000 7f98d552b640 1 -- 172.21.15.71:0/2213267697 shutdown_connections 2024-03-29T11:05:59.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:59.288+0000 7f98d552b640 1 -- 172.21.15.71:0/2213267697 wait complete. 2024-03-29T11:05:59.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:05:59.288+0000 7f98d552b640 1 librados: shutdown 2024-03-29T11:05:59.288 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:06:06.095 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:06:06.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.250+0000 7f87fe515640 10 monclient: get_monmap_and_config 2024-03-29T11:06:06.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.250+0000 7f87fe515640 10 monclient: build_initial_monmap 2024-03-29T11:06:06.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.250+0000 7f87fe515640 10 monclient: monmap: 2024-03-29T11:06:06.230 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:06:06.230 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:06.230 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:06:06.250967+0000 2024-03-29T11:06:06.230 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:06:06.250967+0000 2024-03-29T11:06:06.231 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:06.231 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:06.231 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:06:06.231 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:06:06.231 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:06:06.231 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:06.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.251+0000 7f87fe515640 1 Processor -- start 2024-03-29T11:06:06.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.251+0000 7f87fe515640 1 -- start start 2024-03-29T11:06:06.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.251+0000 7f87fe515640 10 monclient: init 2024-03-29T11:06:06.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.252+0000 7f87fe515640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:06.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.252+0000 7f87fe515640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:06:06.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.252+0000 7f87fe515640 10 monclient(hunting): picked mon.noname-a con 0x7f87f81457f0 addr v1:172.21.15.71:6789/0 2024-03-29T11:06:06.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.252+0000 7f87fe515640 10 monclient(hunting): picked mon.noname-b con 0x7f87f813f650 addr v1:172.21.15.71:6790/0 2024-03-29T11:06:06.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.252+0000 7f87fe515640 10 monclient(hunting): picked mon.noname-c con 0x7f87f80a9d70 addr v1:172.21.15.137:6789/0 2024-03-29T11:06:06.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.252+0000 7f87fe515640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f80aaef0 con 0x7f87f81457f0 2024-03-29T11:06:06.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.252+0000 7f87fe515640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f8145bd0 con 0x7f87f80a9d70 2024-03-29T11:06:06.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.252+0000 7f87fe515640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f8145eb0 con 0x7f87f813f650 2024-03-29T11:06:06.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.252+0000 7f87fe515640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:06.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.252+0000 7f87fe515640 10 monclient(hunting): authenticate will time out at 1122.510339s 2024-03-29T11:06:06.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.252+0000 7f87f7fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f87f81457f0 0x7f87f813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:44258/0 (socket says 172.21.15.71:44258) 2024-03-29T11:06:06.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.252+0000 7f87f7fff640 1 -- 172.21.15.71:0/2158544645 learned_addr learned my addr 172.21.15.71:0/2158544645 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:06:06.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.252+0000 7f87f6ffd640 1 -- 172.21.15.71:0/2158544645 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581193084 0 0) 0x7f87f8145eb0 con 0x7f87f813f650 2024-03-29T11:06:06.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.252+0000 7f87f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:06:06.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.252+0000 7f87f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:06.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.252+0000 7f87f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:06.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.252+0000 7f87f6ffd640 10 monclient(hunting): my global_id is 4463 2024-03-29T11:06:06.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.252+0000 7f87f6ffd640 1 -- 172.21.15.71:0/2158544645 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87e0003520 con 0x7f87f813f650 2024-03-29T11:06:06.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.252+0000 7f87f6ffd640 1 -- 172.21.15.71:0/2158544645 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 640284987 0 0) 0x7f87e0003520 con 0x7f87f813f650 2024-03-29T11:06:06.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.252+0000 7f87f6ffd640 1 -- 172.21.15.71:0/2158544645 >> v1:172.21.15.137:6789/0 conn(0x7f87f80a9d70 legacy=0x7f87f814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:06:06.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.256+0000 7f87f6ffd640 1 -- 172.21.15.71:0/2158544645 >> v1:172.21.15.71:6789/0 conn(0x7f87f81457f0 legacy=0x7f87f813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:06.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.256+0000 7f87f6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:06:06.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.256+0000 7f87f6ffd640 1 monclient: found mon.noname-b 2024-03-29T11:06:06.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.256+0000 7f87f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:06.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.256+0000 7f87f6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:06:06.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.256+0000 7f87f6ffd640 1 -- 172.21.15.71:0/2158544645 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f8146340 con 0x7f87f813f650 2024-03-29T11:06:06.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.256+0000 7f87f6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:06:06.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.256+0000 7f87f6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:06:06.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.256+0000 7f87f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:06.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.256+0000 7f87f6ffd640 1 -- 172.21.15.71:0/2158544645 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f87ec001690 con 0x7f87f813f650 2024-03-29T11:06:06.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.256+0000 7f87fe515640 5 monclient: authenticate success, global_id 4463 2024-03-29T11:06:06.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.256+0000 7f87fe515640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:06:06.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.256+0000 7f87f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:06.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.256+0000 7f87f6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:06:06.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.256+0000 7f87f6ffd640 10 monclient: dump: 2024-03-29T11:06:06.237 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:06.237 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:06.237 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:06.237 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:06.237 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:06.237 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:06.237 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:06.238 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:06.238 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:06.238 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:06.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.256+0000 7f87fe515640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:06:06.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.256+0000 7f87f6ffd640 1 -- 172.21.15.71:0/2158544645 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87ec001860 con 0x7f87f813f650 2024-03-29T11:06:06.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.256+0000 7f87f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:06.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.256+0000 7f87f6ffd640 1 -- 172.21.15.71:0/2158544645 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f87ec001cb0 con 0x7f87f813f650 2024-03-29T11:06:06.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.256+0000 7f87f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:06.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.256+0000 7f87f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:06:06.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.256+0000 7f87f6ffd640 10 monclient: dump: 2024-03-29T11:06:06.239 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:06.239 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:06.239 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:06.239 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:06.239 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:06.239 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:06.239 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:06.239 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:06.239 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:06.239 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:06.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.256+0000 7f87fe515640 10 monclient: get_monmap_and_config success 2024-03-29T11:06:06.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.256+0000 7f87fe515640 4 set_mon_vals no callback set 2024-03-29T11:06:06.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.257+0000 7f87fe515640 10 monclient: shutdown 2024-03-29T11:06:06.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.257+0000 7f87fe515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:06.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.257+0000 7f87fe515640 1 -- 172.21.15.71:0/2158544645 >> v1:172.21.15.71:6790/0 conn(0x7f87f813f650 legacy=0x7f87f80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:06.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.257+0000 7f87fe515640 1 -- 172.21.15.71:0/2158544645 shutdown_connections 2024-03-29T11:06:06.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.257+0000 7f87fe515640 1 -- 172.21.15.71:0/2158544645 >> 172.21.15.71:0/2158544645 conn(0x7f87f8066a00 msgr2=0x7f87f8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:06.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.258+0000 7f87fe515640 1 -- 172.21.15.71:0/2158544645 shutdown_connections 2024-03-29T11:06:06.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.258+0000 7f87fe515640 1 -- 172.21.15.71:0/2158544645 wait complete. 2024-03-29T11:06:06.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.259+0000 7f87fe515640 10 monclient: build_initial_monmap 2024-03-29T11:06:06.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.259+0000 7f87fe515640 10 monclient: monmap: 2024-03-29T11:06:06.244 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:06:06.244 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:06:06.244 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:06:06.244 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:06:06.244 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:06.244 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:06.244 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:06:06.244 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:06:06.245 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:06:06.245 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:06.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.259+0000 7f87fe515640 1 librados: starting msgr at 2024-03-29T11:06:06.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.259+0000 7f87fe515640 1 librados: starting objecter 2024-03-29T11:06:06.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.259+0000 7f87fe515640 1 Processor -- start 2024-03-29T11:06:06.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.259+0000 7f87fe515640 1 -- start start 2024-03-29T11:06:06.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.259+0000 7f87fe515640 1 librados: setting wanted keys 2024-03-29T11:06:06.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.259+0000 7f87fe515640 1 librados: calling monclient init 2024-03-29T11:06:06.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.259+0000 7f87fe515640 10 monclient: init 2024-03-29T11:06:06.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.260+0000 7f87fe515640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:06.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.260+0000 7f87fe515640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:06:06.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.260+0000 7f87fe515640 10 monclient(hunting): picked mon.noname-b con 0x7f87f80a9d70 addr v1:172.21.15.137:6789/0 2024-03-29T11:06:06.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.260+0000 7f87fe515640 10 monclient(hunting): picked mon.noname-a con 0x7f87f813f650 addr v1:172.21.15.71:6789/0 2024-03-29T11:06:06.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.260+0000 7f87f77fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f87f813f650 0x7f87f813fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:44450/0 (socket says 172.21.15.71:44450) 2024-03-29T11:06:06.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.260+0000 7f87f77fe640 1 -- 172.21.15.71:0/3057923709 learned_addr learned my addr 172.21.15.71:0/3057923709 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:06:06.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.260+0000 7f87f4ff9640 10 client.?.objecter ms_handle_connect 0x7f87f813f650 2024-03-29T11:06:06.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.260+0000 7f87f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:06.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.260+0000 7f87fe515640 10 monclient(hunting): picked mon.noname-c con 0x7f87f81457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:06:06.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.261+0000 7f87f4ff9640 10 client.?.objecter ms_handle_connect 0x7f87f81457f0 2024-03-29T11:06:06.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.261+0000 7f87f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:06.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.261+0000 7f87fe515640 1 -- 172.21.15.71:0/3057923709 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f8141b10 con 0x7f87f813f650 2024-03-29T11:06:06.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.261+0000 7f87fe515640 1 -- 172.21.15.71:0/3057923709 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f8141ff0 con 0x7f87f80a9d70 2024-03-29T11:06:06.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.261+0000 7f87fe515640 1 -- 172.21.15.71:0/3057923709 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f8142180 con 0x7f87f81457f0 2024-03-29T11:06:06.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.261+0000 7f87fe515640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:06.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.261+0000 7f87f4ff9640 1 -- 172.21.15.71:0/3057923709 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 574842678 0 0) 0x7f87f8141b10 con 0x7f87f813f650 2024-03-29T11:06:06.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.261+0000 7f87fe515640 10 monclient(hunting): authenticate will time out at 1122.519554s 2024-03-29T11:06:06.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.261+0000 7f87f4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:06:06.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.261+0000 7f87f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:06.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.261+0000 7f87f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:06.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.261+0000 7f87f4ff9640 10 monclient(hunting): my global_id is 4440 2024-03-29T11:06:06.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.261+0000 7f87f4ff9640 1 -- 172.21.15.71:0/3057923709 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87e4004eb0 con 0x7f87f813f650 2024-03-29T11:06:06.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.261+0000 7f87f4ff9640 1 -- 172.21.15.71:0/3057923709 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4091039097 0 0) 0x7f87f8142180 con 0x7f87f81457f0 2024-03-29T11:06:06.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.261+0000 7f87f4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:06:06.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.261+0000 7f87f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:06.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.261+0000 7f87f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:06.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.261+0000 7f87f4ff9640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:06:06.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.261+0000 7f87f4ff9640 1 -- 172.21.15.71:0/3057923709 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87f8141b10 con 0x7f87f81457f0 2024-03-29T11:06:06.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.261+0000 7f87f4ff9640 1 -- 172.21.15.71:0/3057923709 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2494450165 0 0) 0x7f87e4004eb0 con 0x7f87f813f650 2024-03-29T11:06:06.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.261+0000 7f87f4ff9640 1 -- 172.21.15.71:0/3057923709 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87f8142180 con 0x7f87f813f650 2024-03-29T11:06:06.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.261+0000 7f87f4ff9640 1 -- 172.21.15.71:0/3057923709 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f87ec001bb0 con 0x7f87f813f650 2024-03-29T11:06:06.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.261+0000 7f87f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:06.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.261+0000 7f87f4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:06:06.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.261+0000 7f87f4ff9640 10 monclient(hunting): dump: 2024-03-29T11:06:06.248 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:06.248 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:06.248 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:06.248 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:06.248 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:06.248 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:06.248 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:06.248 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:06.248 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:06.248 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:06.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.261+0000 7f87f4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:06:06.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.262+0000 7f87f4ff9640 1 -- 172.21.15.71:0/3057923709 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3135446930 0 0) 0x7f87f8142180 con 0x7f87f813f650 2024-03-29T11:06:06.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.262+0000 7f87f4ff9640 1 -- 172.21.15.71:0/3057923709 >> v1:172.21.15.71:6790/0 conn(0x7f87f81457f0 legacy=0x7f87f81404b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:06.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.262+0000 7f87f4ff9640 1 -- 172.21.15.71:0/3057923709 >> v1:172.21.15.137:6789/0 conn(0x7f87f80a9d70 legacy=0x7f87f8142410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:06:06.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.264+0000 7f87f4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:06:06.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.264+0000 7f87f4ff9640 1 monclient: found mon.a 2024-03-29T11:06:06.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.264+0000 7f87f4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:06.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.264+0000 7f87f4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:06:06.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.264+0000 7f87f4ff9640 1 -- 172.21.15.71:0/3057923709 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f8142b20 con 0x7f87f813f650 2024-03-29T11:06:06.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.264+0000 7f87f4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:06:06.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.264+0000 7f87f4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:06:06.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.264+0000 7f87f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:06.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.264+0000 7f87fe515640 5 monclient: authenticate success, global_id 4440 2024-03-29T11:06:06.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.264+0000 7f87fe515640 10 monclient: _renew_subs 2024-03-29T11:06:06.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.264+0000 7f87fe515640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:06:06.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.264+0000 7f87fe515640 1 -- 172.21.15.71:0/3057923709 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f8071e60 con 0x7f87f813f650 2024-03-29T11:06:06.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.264+0000 7f87fe515640 10 client.4440.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:06:06.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.264+0000 7f87fe515640 10 monclient: _renew_subs 2024-03-29T11:06:06.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.264+0000 7f87fe515640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:06:06.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.264+0000 7f87fe515640 1 -- 172.21.15.71:0/3057923709 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f81398c0 con 0x7f87f813f650 2024-03-29T11:06:06.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.264+0000 7f87fe515640 1 librados: init done 2024-03-29T11:06:06.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.264+0000 7f87f4ff9640 1 -- 172.21.15.71:0/3057923709 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87ec003960 con 0x7f87f813f650 2024-03-29T11:06:06.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.264+0000 7f87f4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:06.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.264+0000 7f87f4ff9640 1 -- 172.21.15.71:0/3057923709 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f87ec001270 con 0x7f87f813f650 2024-03-29T11:06:06.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.264+0000 7f87f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:06.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.264+0000 7f87f4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:06:06.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.264+0000 7f87f4ff9640 10 monclient: dump: 2024-03-29T11:06:06.255 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:06.255 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:06.255 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:06.255 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:06.255 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:06.255 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:06.255 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:06.255 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:06.255 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:06.255 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:06.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.265+0000 7f87f67fc640 4 set_mon_vals no callback set 2024-03-29T11:06:06.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.265+0000 7f87fe515640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:06.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.266+0000 7f87fe515640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:06.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.266+0000 7f87fe515640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:06:06.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.266+0000 7f87fe515640 1 -- 172.21.15.71:0/3057923709 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87d8005080 con 0x7f87f813f650 2024-03-29T11:06:06.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.267+0000 7f87f4ff9640 1 -- 172.21.15.71:0/3057923709 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f87ec003d70 con 0x7f87f813f650 2024-03-29T11:06:06.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.267+0000 7f87f4ff9640 1 -- 172.21.15.71:0/3057923709 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f87ec084b90 con 0x7f87f813f650 2024-03-29T11:06:06.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.267+0000 7f87f4ff9640 10 client.4440.objecter ms_dispatch 0x7f87f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:06:06.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.267+0000 7f87f4ff9640 3 client.4440.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:06:06.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.267+0000 7f87f4ff9640 3 client.4440.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:06:06.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.267+0000 7f87f4ff9640 20 client.4440.objecter dump_active .. 0 homeless 2024-03-29T11:06:06.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.268+0000 7f87f7fff640 10 monclient: get_auth_request con 0x7f87e406dd00 auth_method 0 2024-03-29T11:06:06.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.269+0000 7f87f4ff9640 10 client.4440.objecter ms_handle_connect 0x7f87e406dd00 2024-03-29T11:06:06.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.275+0000 7f87f4ff9640 1 -- 172.21.15.71:0/3057923709 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f87ec050470 con 0x7f87f813f650 2024-03-29T11:06:06.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.275+0000 7f87f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:06.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.275+0000 7f87f4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:06:06.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.591+0000 7f87fe515640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:06.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.591+0000 7f87fe515640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:06.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.591+0000 7f87fe515640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:06:06.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.591+0000 7f87fe515640 1 -- 172.21.15.71:0/3057923709 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87d8005330 con 0x7f87f813f650 2024-03-29T11:06:06.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.592+0000 7f87f4ff9640 1 -- 172.21.15.71:0/3057923709 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f87ec0523a0 con 0x7f87f813f650 2024-03-29T11:06:06.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.592+0000 7f87f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:06.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.592+0000 7f87f4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:06:06.572 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:06:06.572 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:06:06.574 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.596+0000 7f87fe515640 10 librados: watch_flush enter 2024-03-29T11:06:06.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.596+0000 7f87fe515640 10 librados: watch_flush exit 2024-03-29T11:06:06.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.596+0000 7f87fe515640 20 client.4440.objecter shutdown clearing up homeless session... 2024-03-29T11:06:06.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.596+0000 7f87fe515640 10 client.4440.objecter successfully canceled tick 2024-03-29T11:06:06.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.597+0000 7f87fe515640 1 -- 172.21.15.71:0/3057923709 >> v1:172.21.15.137:6816/973332665 conn(0x7f87e406dd00 legacy=0x7f87e40701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:06.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.597+0000 7f87fe515640 10 monclient: shutdown 2024-03-29T11:06:06.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.597+0000 7f87fe515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:06.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.597+0000 7f87fe515640 1 -- 172.21.15.71:0/3057923709 >> v1:172.21.15.71:6789/0 conn(0x7f87f813f650 legacy=0x7f87f813fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:06.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.597+0000 7f87fe515640 1 -- 172.21.15.71:0/3057923709 shutdown_connections 2024-03-29T11:06:06.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.597+0000 7f87fe515640 1 -- 172.21.15.71:0/3057923709 >> 172.21.15.71:0/3057923709 conn(0x7f87f8066a00 msgr2=0x7f87f8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:06.577 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.597+0000 7f87fe515640 1 -- 172.21.15.71:0/3057923709 shutdown_connections 2024-03-29T11:06:06.577 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.597+0000 7f87fe515640 1 -- 172.21.15.71:0/3057923709 wait complete. 2024-03-29T11:06:06.577 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:06.597+0000 7f87fe515640 1 librados: shutdown 2024-03-29T11:06:06.591 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:06:13.397 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:06:13.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.554+0000 7fcb3cf67640 10 monclient: get_monmap_and_config 2024-03-29T11:06:13.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.554+0000 7fcb3cf67640 10 monclient: build_initial_monmap 2024-03-29T11:06:13.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.554+0000 7fcb3cf67640 10 monclient: monmap: 2024-03-29T11:06:13.533 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:06:13.533 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:13.533 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:06:13.555140+0000 2024-03-29T11:06:13.533 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:06:13.555140+0000 2024-03-29T11:06:13.533 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:13.533 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:13.534 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:06:13.534 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:06:13.534 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:06:13.534 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:13.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.555+0000 7fcb3cf67640 1 Processor -- start 2024-03-29T11:06:13.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.555+0000 7fcb3cf67640 1 -- start start 2024-03-29T11:06:13.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.555+0000 7fcb3cf67640 10 monclient: init 2024-03-29T11:06:13.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.556+0000 7fcb3cf67640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:13.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.556+0000 7fcb3cf67640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:06:13.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.556+0000 7fcb3cf67640 10 monclient(hunting): picked mon.noname-c con 0x7fcb380b1f60 addr v1:172.21.15.137:6789/0 2024-03-29T11:06:13.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.556+0000 7fcb3cf67640 10 monclient(hunting): picked mon.noname-a con 0x7fcb38071eb0 addr v1:172.21.15.71:6789/0 2024-03-29T11:06:13.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.556+0000 7fcb3cf67640 10 monclient(hunting): picked mon.noname-b con 0x7fcb38143d10 addr v1:172.21.15.71:6790/0 2024-03-29T11:06:13.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.556+0000 7fcb3cf67640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb3806ef80 con 0x7fcb38071eb0 2024-03-29T11:06:13.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.556+0000 7fcb3cf67640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb3806f260 con 0x7fcb380b1f60 2024-03-29T11:06:13.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.556+0000 7fcb3cf67640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb380b40c0 con 0x7fcb38143d10 2024-03-29T11:06:13.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.556+0000 7fcb3cf67640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:13.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.556+0000 7fcb3cf67640 10 monclient(hunting): authenticate will time out at 1129.814453s 2024-03-29T11:06:13.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.556+0000 7fcb36d9d640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fcb38143d10 0x7fcb38146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:56986/0 (socket says 172.21.15.71:56986) 2024-03-29T11:06:13.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.556+0000 7fcb36d9d640 1 -- 172.21.15.71:0/1488693256 learned_addr learned my addr 172.21.15.71:0/1488693256 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:06:13.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.556+0000 7fcb35d9b640 1 -- 172.21.15.71:0/1488693256 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124357568 0 0) 0x7fcb380b40c0 con 0x7fcb38143d10 2024-03-29T11:06:13.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.557+0000 7fcb35d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:06:13.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.557+0000 7fcb35d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:13.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.557+0000 7fcb35d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:13.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.557+0000 7fcb35d9b640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:06:13.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.557+0000 7fcb35d9b640 1 -- 172.21.15.71:0/1488693256 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb24003520 con 0x7fcb38143d10 2024-03-29T11:06:13.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.557+0000 7fcb35d9b640 1 -- 172.21.15.71:0/1488693256 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 73758975 0 0) 0x7fcb24003520 con 0x7fcb38143d10 2024-03-29T11:06:13.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.557+0000 7fcb35d9b640 1 -- 172.21.15.71:0/1488693256 >> v1:172.21.15.137:6789/0 conn(0x7fcb380b1f60 legacy=0x7fcb38143600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:06:13.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.557+0000 7fcb35d9b640 1 -- 172.21.15.71:0/1488693256 >> v1:172.21.15.71:6789/0 conn(0x7fcb38071eb0 legacy=0x7fcb38146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:06:13.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.558+0000 7fcb35d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:06:13.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.558+0000 7fcb35d9b640 1 monclient: found mon.noname-b 2024-03-29T11:06:13.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.558+0000 7fcb35d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:13.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.558+0000 7fcb35d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:06:13.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.558+0000 7fcb35d9b640 1 -- 172.21.15.71:0/1488693256 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb38155d70 con 0x7fcb38143d10 2024-03-29T11:06:13.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.558+0000 7fcb35d9b640 10 monclient: _finish_auth 0 2024-03-29T11:06:13.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.558+0000 7fcb35d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:06:13.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.558+0000 7fcb35d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:13.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.558+0000 7fcb35d9b640 1 -- 172.21.15.71:0/1488693256 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fcb30001a10 con 0x7fcb38143d10 2024-03-29T11:06:13.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.558+0000 7fcb35d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:13.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.558+0000 7fcb35d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:06:13.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.558+0000 7fcb35d9b640 10 monclient: dump: 2024-03-29T11:06:13.538 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:13.538 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:13.538 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:13.538 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:13.538 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:13.538 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:13.538 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:13.538 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:13.539 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:13.539 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:13.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.559+0000 7fcb3cf67640 5 monclient: authenticate success, global_id 4469 2024-03-29T11:06:13.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.559+0000 7fcb3cf67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:06:13.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.559+0000 7fcb35d9b640 1 -- 172.21.15.71:0/1488693256 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb30001be0 con 0x7fcb38143d10 2024-03-29T11:06:13.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.559+0000 7fcb35d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:13.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.559+0000 7fcb3cf67640 10 monclient: get_monmap_and_config success 2024-03-29T11:06:13.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.559+0000 7fcb3cf67640 4 set_mon_vals no callback set 2024-03-29T11:06:13.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.560+0000 7fcb3cf67640 10 monclient: shutdown 2024-03-29T11:06:13.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.560+0000 7fcb3cf67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:13.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.560+0000 7fcb3cf67640 1 -- 172.21.15.71:0/1488693256 >> v1:172.21.15.71:6790/0 conn(0x7fcb38143d10 legacy=0x7fcb38146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:13.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.560+0000 7fcb3cf67640 1 -- 172.21.15.71:0/1488693256 shutdown_connections 2024-03-29T11:06:13.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.560+0000 7fcb3cf67640 1 -- 172.21.15.71:0/1488693256 >> 172.21.15.71:0/1488693256 conn(0x7fcb38066a00 msgr2=0x7fcb38066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:13.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.560+0000 7fcb3cf67640 1 -- 172.21.15.71:0/1488693256 shutdown_connections 2024-03-29T11:06:13.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.560+0000 7fcb3cf67640 1 -- 172.21.15.71:0/1488693256 wait complete. 2024-03-29T11:06:13.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.561+0000 7fcb3cf67640 10 monclient: build_initial_monmap 2024-03-29T11:06:13.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.561+0000 7fcb3cf67640 10 monclient: monmap: 2024-03-29T11:06:13.542 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:06:13.542 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:06:13.542 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:06:13.542 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:06:13.542 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:13.542 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:13.542 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:06:13.542 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:06:13.542 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:06:13.542 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:13.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.561+0000 7fcb3cf67640 1 librados: starting msgr at 2024-03-29T11:06:13.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.561+0000 7fcb3cf67640 1 librados: starting objecter 2024-03-29T11:06:13.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.562+0000 7fcb3cf67640 1 Processor -- start 2024-03-29T11:06:13.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.562+0000 7fcb3cf67640 1 -- start start 2024-03-29T11:06:13.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.562+0000 7fcb3cf67640 1 librados: setting wanted keys 2024-03-29T11:06:13.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.562+0000 7fcb3cf67640 1 librados: calling monclient init 2024-03-29T11:06:13.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.562+0000 7fcb3cf67640 10 monclient: init 2024-03-29T11:06:13.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.562+0000 7fcb3cf67640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:13.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.562+0000 7fcb3cf67640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:06:13.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.562+0000 7fcb3cf67640 10 monclient(hunting): picked mon.noname-c con 0x7fcb380b1f60 addr v1:172.21.15.71:6790/0 2024-03-29T11:06:13.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.562+0000 7fcb3659c640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fcb380b1f60 0x7fcb380b7230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:57064/0 (socket says 172.21.15.71:57064) 2024-03-29T11:06:13.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.562+0000 7fcb3659c640 1 -- 172.21.15.71:0/1751962586 learned_addr learned my addr 172.21.15.71:0/1751962586 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:06:13.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.563+0000 7fcb3cf67640 10 monclient(hunting): picked mon.noname-b con 0x7fcb38143d10 addr v1:172.21.15.137:6789/0 2024-03-29T11:06:13.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.563+0000 7fcb2f7fe640 10 client.?.objecter ms_handle_connect 0x7fcb380b1f60 2024-03-29T11:06:13.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.563+0000 7fcb2f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:13.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.563+0000 7fcb3cf67640 10 monclient(hunting): picked mon.noname-a con 0x7fcb381547c0 addr v1:172.21.15.71:6789/0 2024-03-29T11:06:13.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.563+0000 7fcb3cf67640 1 -- 172.21.15.71:0/1751962586 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb380b8880 con 0x7fcb381547c0 2024-03-29T11:06:13.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.563+0000 7fcb3cf67640 1 -- 172.21.15.71:0/1751962586 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb38139240 con 0x7fcb38143d10 2024-03-29T11:06:13.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.563+0000 7fcb2f7fe640 10 client.?.objecter ms_handle_connect 0x7fcb38143d10 2024-03-29T11:06:13.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.563+0000 7fcb2f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:13.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.563+0000 7fcb3cf67640 1 -- 172.21.15.71:0/1751962586 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb380b7c20 con 0x7fcb380b1f60 2024-03-29T11:06:13.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.563+0000 7fcb3cf67640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:13.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.563+0000 7fcb3cf67640 10 monclient(hunting): authenticate will time out at 1129.821895s 2024-03-29T11:06:13.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.564+0000 7fcb2f7fe640 1 -- 172.21.15.71:0/1751962586 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462232487 0 0) 0x7fcb380b7c20 con 0x7fcb380b1f60 2024-03-29T11:06:13.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.564+0000 7fcb2f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:06:13.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.564+0000 7fcb2f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:13.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.564+0000 7fcb2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:13.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.564+0000 7fcb2f7fe640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:06:13.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.564+0000 7fcb2f7fe640 1 -- 172.21.15.71:0/1751962586 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb18004eb0 con 0x7fcb380b1f60 2024-03-29T11:06:13.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.564+0000 7fcb2f7fe640 1 -- 172.21.15.71:0/1751962586 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 941386642 0 0) 0x7fcb38139240 con 0x7fcb38143d10 2024-03-29T11:06:13.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.564+0000 7fcb2f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:06:13.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.564+0000 7fcb2f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:13.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.564+0000 7fcb2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:13.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.564+0000 7fcb2f7fe640 10 monclient(hunting): my global_id is 4408 2024-03-29T11:06:13.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.564+0000 7fcb2f7fe640 1 -- 172.21.15.71:0/1751962586 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb380b7c20 con 0x7fcb38143d10 2024-03-29T11:06:13.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.564+0000 7fcb2f7fe640 1 -- 172.21.15.71:0/1751962586 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3089004651 0 0) 0x7fcb18004eb0 con 0x7fcb380b1f60 2024-03-29T11:06:13.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.564+0000 7fcb2f7fe640 1 -- 172.21.15.71:0/1751962586 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb38139240 con 0x7fcb380b1f60 2024-03-29T11:06:13.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.565+0000 7fcb2f7fe640 1 -- 172.21.15.71:0/1751962586 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fcb20001870 con 0x7fcb380b1f60 2024-03-29T11:06:13.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.565+0000 7fcb2f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:13.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.565+0000 7fcb2f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:06:13.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.565+0000 7fcb2f7fe640 10 monclient(hunting): dump: 2024-03-29T11:06:13.546 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:13.546 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:13.546 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:13.546 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:13.546 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:13.546 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:13.546 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:13.547 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:13.547 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:13.547 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:13.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.565+0000 7fcb2f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:06:13.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.565+0000 7fcb2f7fe640 1 -- 172.21.15.71:0/1751962586 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2770550736 0 0) 0x7fcb380b7c20 con 0x7fcb38143d10 2024-03-29T11:06:13.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.565+0000 7fcb2f7fe640 1 -- 172.21.15.71:0/1751962586 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb18004eb0 con 0x7fcb38143d10 2024-03-29T11:06:13.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.565+0000 7fcb2f7fe640 1 -- 172.21.15.71:0/1751962586 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fcb28001830 con 0x7fcb38143d10 2024-03-29T11:06:13.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.565+0000 7fcb2f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:13.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.565+0000 7fcb2f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:06:13.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.565+0000 7fcb2f7fe640 10 monclient(hunting): dump: 2024-03-29T11:06:13.547 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:13.547 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:13.547 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:13.547 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:13.548 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:13.548 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:13.548 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:13.548 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:13.548 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:13.548 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:13.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.565+0000 7fcb2f7fe640 1 -- 172.21.15.71:0/1751962586 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3572661690 0 0) 0x7fcb38139240 con 0x7fcb380b1f60 2024-03-29T11:06:13.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.565+0000 7fcb2f7fe640 1 -- 172.21.15.71:0/1751962586 >> v1:172.21.15.137:6789/0 conn(0x7fcb38143d10 legacy=0x7fcb381540b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:13.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.565+0000 7fcb2f7fe640 1 -- 172.21.15.71:0/1751962586 >> v1:172.21.15.71:6789/0 conn(0x7fcb381547c0 legacy=0x7fcb381e5ea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:06:13.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.567+0000 7fcb2f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:06:13.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.567+0000 7fcb2f7fe640 1 monclient: found mon.c 2024-03-29T11:06:13.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.567+0000 7fcb2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:13.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.567+0000 7fcb2f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:06:13.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.567+0000 7fcb2f7fe640 1 -- 172.21.15.71:0/1751962586 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb380b7940 con 0x7fcb380b1f60 2024-03-29T11:06:13.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.567+0000 7fcb2f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:06:13.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.567+0000 7fcb2f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:06:13.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.567+0000 7fcb2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:13.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.567+0000 7fcb3cf67640 5 monclient: authenticate success, global_id 4472 2024-03-29T11:06:13.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.567+0000 7fcb3cf67640 10 monclient: _renew_subs 2024-03-29T11:06:13.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.567+0000 7fcb3cf67640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:06:13.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.567+0000 7fcb3cf67640 1 -- 172.21.15.71:0/1751962586 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb380b67e0 con 0x7fcb380b1f60 2024-03-29T11:06:13.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.567+0000 7fcb3cf67640 10 client.4472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:06:13.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.567+0000 7fcb3cf67640 10 monclient: _renew_subs 2024-03-29T11:06:13.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.567+0000 7fcb3cf67640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:06:13.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.567+0000 7fcb3cf67640 1 -- 172.21.15.71:0/1751962586 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb381e75c0 con 0x7fcb380b1f60 2024-03-29T11:06:13.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.567+0000 7fcb2f7fe640 1 -- 172.21.15.71:0/1751962586 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb20001d80 con 0x7fcb380b1f60 2024-03-29T11:06:13.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.567+0000 7fcb2f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:13.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.567+0000 7fcb2f7fe640 1 -- 172.21.15.71:0/1751962586 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fcb20004ac0 con 0x7fcb380b1f60 2024-03-29T11:06:13.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.567+0000 7fcb2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:13.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.567+0000 7fcb2f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:06:13.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.567+0000 7fcb2f7fe640 10 monclient: dump: 2024-03-29T11:06:13.556 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:13.556 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:13.556 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:13.557 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:13.557 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:13.557 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:13.557 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:13.557 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:13.557 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:13.557 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:13.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.567+0000 7fcb3559a640 4 set_mon_vals no callback set 2024-03-29T11:06:13.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.568+0000 7fcb3cf67640 1 librados: init done 2024-03-29T11:06:13.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.569+0000 7fcb2f7fe640 1 -- 172.21.15.71:0/1751962586 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7fcb2001aa90 con 0x7fcb380b1f60 2024-03-29T11:06:13.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.569+0000 7fcb3cf67640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:13.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.569+0000 7fcb3cf67640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:13.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.569+0000 7fcb3cf67640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:06:13.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.569+0000 7fcb3cf67640 1 -- 172.21.15.71:0/1751962586 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb38075f10 con 0x7fcb380b1f60 2024-03-29T11:06:13.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.570+0000 7fcb2f7fe640 1 -- 172.21.15.71:0/1751962586 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7fcb20084a60 con 0x7fcb380b1f60 2024-03-29T11:06:13.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.570+0000 7fcb2f7fe640 10 client.4472.objecter ms_dispatch 0x7fcb38000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:06:13.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.570+0000 7fcb2f7fe640 3 client.4472.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:06:13.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.570+0000 7fcb2f7fe640 3 client.4472.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:06:13.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.570+0000 7fcb2f7fe640 20 client.4472.objecter dump_active .. 0 homeless 2024-03-29T11:06:13.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.570+0000 7fcb2dd9b640 10 monclient: get_auth_request con 0x7fcb1806ded0 auth_method 0 2024-03-29T11:06:13.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.570+0000 7fcb2f7fe640 10 client.4472.objecter ms_handle_connect 0x7fcb1806ded0 2024-03-29T11:06:13.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.576+0000 7fcb2f7fe640 1 -- 172.21.15.71:0/1751962586 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcb20050340 con 0x7fcb380b1f60 2024-03-29T11:06:13.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.576+0000 7fcb2f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:13.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.576+0000 7fcb2f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:06:13.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.882+0000 7fcb3cf67640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:13.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.882+0000 7fcb3cf67640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:13.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.882+0000 7fcb3cf67640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:06:13.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.882+0000 7fcb3cf67640 1 -- 172.21.15.71:0/1751962586 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb38074f50 con 0x7fcb380b1f60 2024-03-29T11:06:13.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.883+0000 7fcb2f7fe640 1 -- 172.21.15.71:0/1751962586 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7fcb20052270 con 0x7fcb380b1f60 2024-03-29T11:06:13.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.883+0000 7fcb2f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:13.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.883+0000 7fcb2f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:06:13.863 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:06:13.863 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:06:13.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.886+0000 7fcb3cf67640 10 librados: watch_flush enter 2024-03-29T11:06:13.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.886+0000 7fcb3cf67640 10 librados: watch_flush exit 2024-03-29T11:06:13.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.886+0000 7fcb3cf67640 20 client.4472.objecter shutdown clearing up homeless session... 2024-03-29T11:06:13.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.886+0000 7fcb3cf67640 10 client.4472.objecter successfully canceled tick 2024-03-29T11:06:13.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.886+0000 7fcb3cf67640 1 -- 172.21.15.71:0/1751962586 >> v1:172.21.15.137:6816/973332665 conn(0x7fcb1806ded0 legacy=0x7fcb18070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:13.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.886+0000 7fcb3cf67640 10 monclient: shutdown 2024-03-29T11:06:13.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.886+0000 7fcb3cf67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:13.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.886+0000 7fcb3cf67640 1 -- 172.21.15.71:0/1751962586 >> v1:172.21.15.71:6790/0 conn(0x7fcb380b1f60 legacy=0x7fcb380b7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:13.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.886+0000 7fcb3cf67640 1 -- 172.21.15.71:0/1751962586 shutdown_connections 2024-03-29T11:06:13.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.886+0000 7fcb3cf67640 1 -- 172.21.15.71:0/1751962586 >> 172.21.15.71:0/1751962586 conn(0x7fcb38066a00 msgr2=0x7fcb38066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:13.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.886+0000 7fcb3cf67640 1 -- 172.21.15.71:0/1751962586 shutdown_connections 2024-03-29T11:06:13.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.886+0000 7fcb3cf67640 1 -- 172.21.15.71:0/1751962586 wait complete. 2024-03-29T11:06:13.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:13.886+0000 7fcb3cf67640 1 librados: shutdown 2024-03-29T11:06:13.878 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:06:20.684 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:06:20.830 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.850+0000 7f136b463640 10 monclient: get_monmap_and_config 2024-03-29T11:06:20.830 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.850+0000 7f136b463640 10 monclient: build_initial_monmap 2024-03-29T11:06:20.830 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.850+0000 7f136b463640 10 monclient: monmap: 2024-03-29T11:06:20.830 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:06:20.830 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:20.830 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:06:20.851603+0000 2024-03-29T11:06:20.830 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:06:20.851603+0000 2024-03-29T11:06:20.830 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:20.830 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:20.830 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:06:20.830 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:06:20.830 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:06:20.830 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:20.831 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.851+0000 7f136b463640 1 Processor -- start 2024-03-29T11:06:20.831 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.852+0000 7f136b463640 1 -- start start 2024-03-29T11:06:20.831 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.852+0000 7f136b463640 10 monclient: init 2024-03-29T11:06:20.831 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.853+0000 7f136b463640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:20.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.853+0000 7f136b463640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:06:20.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.853+0000 7f136b463640 10 monclient(hunting): picked mon.noname-a con 0x7f13640b4920 addr v1:172.21.15.71:6789/0 2024-03-29T11:06:20.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.855+0000 7f136b463640 10 monclient(hunting): picked mon.noname-c con 0x7f13640b1790 addr v1:172.21.15.137:6789/0 2024-03-29T11:06:20.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.857+0000 7f136b463640 10 monclient(hunting): picked mon.noname-b con 0x7f1364146790 addr v1:172.21.15.71:6790/0 2024-03-29T11:06:20.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.857+0000 7f136b463640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13640b4070 con 0x7f13640b4920 2024-03-29T11:06:20.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.857+0000 7f136b463640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f136406fb80 con 0x7f13640b1790 2024-03-29T11:06:20.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.857+0000 7f136b463640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13640b42c0 con 0x7f1364146790 2024-03-29T11:06:20.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.857+0000 7f136b463640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:20.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.857+0000 7f136b463640 10 monclient(hunting): authenticate will time out at 1137.115324s 2024-03-29T11:06:20.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.856+0000 7f13691ff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f13640b4920 0x7f136406e510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:56509/0 (socket says 172.21.15.71:56509) 2024-03-29T11:06:20.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.857+0000 7f13691ff640 1 -- 172.21.15.71:0/1338651546 learned_addr learned my addr 172.21.15.71:0/1338651546 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:06:20.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.858+0000 7f1363fff640 1 -- 172.21.15.71:0/1338651546 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706113760 0 0) 0x7f13640b4070 con 0x7f13640b4920 2024-03-29T11:06:20.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.858+0000 7f1363fff640 10 monclient(hunting): _negotiate 2024-03-29T11:06:20.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.858+0000 7f1363fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:20.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.858+0000 7f1363fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:20.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.858+0000 7f1363fff640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:06:20.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.858+0000 7f1363fff640 1 -- 172.21.15.71:0/1338651546 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f134c003520 con 0x7f13640b4920 2024-03-29T11:06:20.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.858+0000 7f1363fff640 1 -- 172.21.15.71:0/1338651546 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4119606670 0 0) 0x7f13640b42c0 con 0x7f1364146790 2024-03-29T11:06:20.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.858+0000 7f1363fff640 10 monclient(hunting): _negotiate 2024-03-29T11:06:20.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.858+0000 7f1363fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:20.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.858+0000 7f1363fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:20.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.858+0000 7f1363fff640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:06:20.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.858+0000 7f1363fff640 1 -- 172.21.15.71:0/1338651546 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f134c003840 con 0x7f1364146790 2024-03-29T11:06:20.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.858+0000 7f1363fff640 1 -- 172.21.15.71:0/1338651546 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2268540692 0 0) 0x7f134c003520 con 0x7f13640b4920 2024-03-29T11:06:20.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.858+0000 7f1363fff640 1 -- 172.21.15.71:0/1338651546 >> v1:172.21.15.71:6790/0 conn(0x7f1364146790 legacy=0x7f1364146c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:20.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.858+0000 7f1363fff640 1 -- 172.21.15.71:0/1338651546 >> v1:172.21.15.137:6789/0 conn(0x7f13640b1790 legacy=0x7f13640b1bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:06:20.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.859+0000 7f1363fff640 10 monclient: _finish_hunting 0 2024-03-29T11:06:20.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.859+0000 7f1363fff640 1 monclient: found mon.noname-a 2024-03-29T11:06:20.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.859+0000 7f1363fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:20.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.859+0000 7f1363fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:06:20.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.859+0000 7f1363fff640 1 -- 172.21.15.71:0/1338651546 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13640b22c0 con 0x7f13640b4920 2024-03-29T11:06:20.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.859+0000 7f1363fff640 10 monclient: _finish_auth 0 2024-03-29T11:06:20.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.859+0000 7f1363fff640 10 monclient: _check_auth_tickets 2024-03-29T11:06:20.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.859+0000 7f1363fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:20.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.859+0000 7f1363fff640 1 -- 172.21.15.71:0/1338651546 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f1354001670 con 0x7f13640b4920 2024-03-29T11:06:20.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.859+0000 7f1363fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:20.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.859+0000 7f1363fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:06:20.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.859+0000 7f1363fff640 10 monclient: dump: 2024-03-29T11:06:20.842 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:20.842 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:20.842 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:20.842 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:20.842 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:20.842 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:20.842 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:20.842 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:20.842 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:20.843 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:20.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.859+0000 7f136b463640 5 monclient: authenticate success, global_id 4443 2024-03-29T11:06:20.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.859+0000 7f136b463640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:06:20.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.859+0000 7f1363fff640 1 -- 172.21.15.71:0/1338651546 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1354001850 con 0x7f13640b4920 2024-03-29T11:06:20.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.859+0000 7f1363fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:20.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.859+0000 7f136b463640 10 monclient: get_monmap_and_config success 2024-03-29T11:06:20.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.859+0000 7f136b463640 4 set_mon_vals no callback set 2024-03-29T11:06:20.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.859+0000 7f1363fff640 1 -- 172.21.15.71:0/1338651546 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f1354001c60 con 0x7f13640b4920 2024-03-29T11:06:20.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.859+0000 7f1363fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:20.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.860+0000 7f1363fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:06:20.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.860+0000 7f1363fff640 10 monclient: dump: 2024-03-29T11:06:20.843 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:20.843 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:20.843 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:20.843 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:20.843 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:20.844 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:20.844 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:20.844 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:20.844 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:20.844 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:20.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.860+0000 7f136b463640 10 monclient: shutdown 2024-03-29T11:06:20.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.860+0000 7f136b463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:20.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.860+0000 7f136b463640 1 -- 172.21.15.71:0/1338651546 >> v1:172.21.15.71:6789/0 conn(0x7f13640b4920 legacy=0x7f136406e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:20.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.861+0000 7f136b463640 1 -- 172.21.15.71:0/1338651546 shutdown_connections 2024-03-29T11:06:20.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.862+0000 7f136b463640 1 -- 172.21.15.71:0/1338651546 >> 172.21.15.71:0/1338651546 conn(0x7f1364066a00 msgr2=0x7f1364066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:20.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.862+0000 7f136b463640 1 -- 172.21.15.71:0/1338651546 shutdown_connections 2024-03-29T11:06:20.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.862+0000 7f136b463640 1 -- 172.21.15.71:0/1338651546 wait complete. 2024-03-29T11:06:20.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.863+0000 7f136b463640 10 monclient: build_initial_monmap 2024-03-29T11:06:20.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.863+0000 7f136b463640 10 monclient: monmap: 2024-03-29T11:06:20.846 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:06:20.846 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:06:20.846 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:06:20.846 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:06:20.846 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:20.846 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:20.846 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:06:20.846 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:06:20.846 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:06:20.846 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:20.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.865+0000 7f136b463640 1 librados: starting msgr at 2024-03-29T11:06:20.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.865+0000 7f136b463640 1 librados: starting objecter 2024-03-29T11:06:20.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.865+0000 7f136b463640 1 Processor -- start 2024-03-29T11:06:20.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.865+0000 7f136b463640 1 -- start start 2024-03-29T11:06:20.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.865+0000 7f136b463640 1 librados: setting wanted keys 2024-03-29T11:06:20.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.865+0000 7f136b463640 1 librados: calling monclient init 2024-03-29T11:06:20.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.865+0000 7f136b463640 10 monclient: init 2024-03-29T11:06:20.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.866+0000 7f136b463640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:20.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.866+0000 7f136b463640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:06:20.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.866+0000 7f136b463640 10 monclient(hunting): picked mon.noname-a con 0x7f13640b1790 addr v1:172.21.15.71:6789/0 2024-03-29T11:06:20.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.866+0000 7f136b463640 10 monclient(hunting): picked mon.noname-b con 0x7f13640b4920 addr v1:172.21.15.137:6789/0 2024-03-29T11:06:20.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.869+0000 7f136b463640 10 monclient(hunting): picked mon.noname-c con 0x7f1364146790 addr v1:172.21.15.71:6790/0 2024-03-29T11:06:20.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.869+0000 7f13691ff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f13640b1790 0x7f13641de950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:56847/0 (socket says 172.21.15.71:56847) 2024-03-29T11:06:20.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.869+0000 7f13691ff640 1 -- 172.21.15.71:0/833501781 learned_addr learned my addr 172.21.15.71:0/833501781 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:06:20.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f136b463640 1 -- 172.21.15.71:0/833501781 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1364134f20 con 0x7f13640b1790 2024-03-29T11:06:20.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f136b463640 1 -- 172.21.15.71:0/833501781 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1364137680 con 0x7f13640b4920 2024-03-29T11:06:20.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f136b463640 1 -- 172.21.15.71:0/833501781 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1364137960 con 0x7f1364146790 2024-03-29T11:06:20.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f136b463640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:20.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f136b463640 10 monclient(hunting): authenticate will time out at 1137.128270s 2024-03-29T11:06:20.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 10 client.?.objecter ms_handle_connect 0x7f13640b1790 2024-03-29T11:06:20.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:20.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 10 client.?.objecter ms_handle_connect 0x7f1364146790 2024-03-29T11:06:20.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:20.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 1 -- 172.21.15.71:0/833501781 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1852306296 0 0) 0x7f1364134f20 con 0x7f13640b1790 2024-03-29T11:06:20.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:06:20.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:20.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:20.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 10 monclient(hunting): my global_id is 4446 2024-03-29T11:06:20.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 1 -- 172.21.15.71:0/833501781 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1350004eb0 con 0x7f13640b1790 2024-03-29T11:06:20.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 1 -- 172.21.15.71:0/833501781 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510968643 0 0) 0x7f1364137960 con 0x7f1364146790 2024-03-29T11:06:20.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:06:20.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:20.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:20.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:06:20.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 1 -- 172.21.15.71:0/833501781 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1364134f20 con 0x7f1364146790 2024-03-29T11:06:20.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 1 -- 172.21.15.71:0/833501781 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 173731263 0 0) 0x7f1350004eb0 con 0x7f13640b1790 2024-03-29T11:06:20.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 1 -- 172.21.15.71:0/833501781 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1364137960 con 0x7f13640b1790 2024-03-29T11:06:20.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 1 -- 172.21.15.71:0/833501781 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f1354001c60 con 0x7f13640b1790 2024-03-29T11:06:20.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:20.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:06:20.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 10 monclient(hunting): dump: 2024-03-29T11:06:20.852 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:20.852 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:20.853 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:20.853 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:20.853 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:20.853 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:20.853 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:20.853 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:20.853 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:20.853 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:20.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:06:20.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 1 -- 172.21.15.71:0/833501781 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3260033376 0 0) 0x7f1364134f20 con 0x7f1364146790 2024-03-29T11:06:20.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 1 -- 172.21.15.71:0/833501781 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1350004eb0 con 0x7f1364146790 2024-03-29T11:06:20.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 1 -- 172.21.15.71:0/833501781 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f135c001b50 con 0x7f1364146790 2024-03-29T11:06:20.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:20.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:06:20.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.870+0000 7f1361ffb640 10 monclient(hunting): dump: 2024-03-29T11:06:20.854 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:20.854 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:20.854 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:20.854 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:20.854 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:20.854 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:20.854 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:20.854 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:20.854 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:20.854 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:20.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.871+0000 7f1361ffb640 1 -- 172.21.15.71:0/833501781 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1941787809 0 0) 0x7f1364137960 con 0x7f13640b1790 2024-03-29T11:06:20.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.871+0000 7f1361ffb640 1 -- 172.21.15.71:0/833501781 >> v1:172.21.15.71:6790/0 conn(0x7f1364146790 legacy=0x7f13641347e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:20.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.871+0000 7f1361ffb640 1 -- 172.21.15.71:0/833501781 >> v1:172.21.15.137:6789/0 conn(0x7f13640b4920 legacy=0x7f13641340d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:06:20.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.873+0000 7f1361ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:06:20.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.874+0000 7f1361ffb640 1 monclient: found mon.a 2024-03-29T11:06:20.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.874+0000 7f1361ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:20.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.874+0000 7f1361ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:06:20.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.874+0000 7f1361ffb640 1 -- 172.21.15.71:0/833501781 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13641361d0 con 0x7f13640b1790 2024-03-29T11:06:20.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.874+0000 7f1361ffb640 10 monclient: _finish_auth 0 2024-03-29T11:06:20.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.874+0000 7f1361ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:06:20.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.874+0000 7f1361ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:20.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.875+0000 7f136b463640 5 monclient: authenticate success, global_id 4446 2024-03-29T11:06:20.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.875+0000 7f136b463640 10 monclient: _renew_subs 2024-03-29T11:06:20.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.875+0000 7f136b463640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:06:20.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.875+0000 7f136b463640 1 -- 172.21.15.71:0/833501781 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13641366d0 con 0x7f13640b1790 2024-03-29T11:06:20.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.875+0000 7f136b463640 10 client.4446.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:06:20.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.875+0000 7f136b463640 10 monclient: _renew_subs 2024-03-29T11:06:20.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.875+0000 7f136b463640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:06:20.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.875+0000 7f136b463640 1 -- 172.21.15.71:0/833501781 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1364135830 con 0x7f13640b1790 2024-03-29T11:06:20.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.875+0000 7f136b463640 1 librados: init done 2024-03-29T11:06:20.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.877+0000 7f134b7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:20.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.877+0000 7f134b7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:20.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.877+0000 7f134b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:06:20.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.877+0000 7f134b7fe640 1 -- 172.21.15.71:0/833501781 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1344005080 con 0x7f13640b1790 2024-03-29T11:06:20.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.877+0000 7f1361ffb640 1 -- 172.21.15.71:0/833501781 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1354004b80 con 0x7f13640b1790 2024-03-29T11:06:20.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.877+0000 7f1361ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:20.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.877+0000 7f1361ffb640 1 -- 172.21.15.71:0/833501781 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f1354004e90 con 0x7f13640b1790 2024-03-29T11:06:20.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.877+0000 7f1361ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:20.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.878+0000 7f1361ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:06:20.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.878+0000 7f1361ffb640 10 monclient: dump: 2024-03-29T11:06:20.858 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:20.858 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:20.858 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:20.858 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:20.858 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:20.858 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:20.859 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:20.859 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:20.859 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:20.859 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:20.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.878+0000 7f1361ffb640 1 -- 172.21.15.71:0/833501781 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f13540050b0 con 0x7f13640b1790 2024-03-29T11:06:20.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.878+0000 7f1361ffb640 1 -- 172.21.15.71:0/833501781 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f1354084ed0 con 0x7f13640b1790 2024-03-29T11:06:20.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.878+0000 7f1361ffb640 10 client.4446.objecter ms_dispatch 0x7f1364000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:06:20.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.878+0000 7f1361ffb640 3 client.4446.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:06:20.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.879+0000 7f1361ffb640 3 client.4446.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:06:20.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.879+0000 7f1361ffb640 20 client.4446.objecter dump_active .. 0 homeless 2024-03-29T11:06:20.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.879+0000 7f13637fe640 4 set_mon_vals no callback set 2024-03-29T11:06:20.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.879+0000 7f13689fe640 10 monclient: get_auth_request con 0x7f135006e100 auth_method 0 2024-03-29T11:06:20.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.884+0000 7f1361ffb640 10 client.4446.objecter ms_handle_connect 0x7f135006e100 2024-03-29T11:06:20.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.887+0000 7f1361ffb640 1 -- 172.21.15.71:0/833501781 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1354050830 con 0x7f13640b1790 2024-03-29T11:06:20.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.887+0000 7f1361ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:20.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:20.887+0000 7f1361ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:06:21.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:21.207+0000 7f134b7fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:21.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:21.207+0000 7f134b7fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:21.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:21.207+0000 7f134b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:06:21.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:21.207+0000 7f134b7fe640 1 -- 172.21.15.71:0/833501781 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1344005330 con 0x7f13640b1790 2024-03-29T11:06:21.187 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:06:21.187 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:06:21.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:21.208+0000 7f1361ffb640 1 -- 172.21.15.71:0/833501781 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f1354052760 con 0x7f13640b1790 2024-03-29T11:06:21.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:21.208+0000 7f1361ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:21.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:21.208+0000 7f1361ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:06:21.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:21.211+0000 7f134b7fe640 10 librados: watch_flush enter 2024-03-29T11:06:21.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:21.211+0000 7f134b7fe640 10 librados: watch_flush exit 2024-03-29T11:06:21.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:21.211+0000 7f134b7fe640 20 client.4446.objecter shutdown clearing up homeless session... 2024-03-29T11:06:21.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:21.211+0000 7f134b7fe640 10 client.4446.objecter successfully canceled tick 2024-03-29T11:06:21.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:21.211+0000 7f134b7fe640 1 -- 172.21.15.71:0/833501781 >> v1:172.21.15.137:6816/973332665 conn(0x7f135006e100 legacy=0x7f13500705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:21.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:21.211+0000 7f134b7fe640 10 monclient: shutdown 2024-03-29T11:06:21.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:21.211+0000 7f134b7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:21.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:21.211+0000 7f134b7fe640 1 -- 172.21.15.71:0/833501781 >> v1:172.21.15.71:6789/0 conn(0x7f13640b1790 legacy=0x7f13641de950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:21.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:21.212+0000 7f134b7fe640 1 -- 172.21.15.71:0/833501781 shutdown_connections 2024-03-29T11:06:21.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:21.212+0000 7f134b7fe640 1 -- 172.21.15.71:0/833501781 >> 172.21.15.71:0/833501781 conn(0x7f1364066a00 msgr2=0x7f136406ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:21.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:21.212+0000 7f134b7fe640 1 -- 172.21.15.71:0/833501781 shutdown_connections 2024-03-29T11:06:21.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:21.212+0000 7f134b7fe640 1 -- 172.21.15.71:0/833501781 wait complete. 2024-03-29T11:06:21.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:21.213+0000 7f134b7fe640 1 librados: shutdown 2024-03-29T11:06:21.211 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:06:28.016 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:06:28.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.183+0000 7f45f669d640 10 monclient: get_monmap_and_config 2024-03-29T11:06:28.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.183+0000 7f45f669d640 10 monclient: build_initial_monmap 2024-03-29T11:06:28.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.183+0000 7f45f669d640 10 monclient: monmap: 2024-03-29T11:06:28.162 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:06:28.162 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:28.162 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:06:28.184009+0000 2024-03-29T11:06:28.162 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:06:28.184009+0000 2024-03-29T11:06:28.162 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:28.162 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:28.163 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:06:28.163 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:06:28.163 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:06:28.163 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:28.163 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.184+0000 7f45f669d640 1 Processor -- start 2024-03-29T11:06:28.163 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.184+0000 7f45f669d640 1 -- start start 2024-03-29T11:06:28.163 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.184+0000 7f45f669d640 10 monclient: init 2024-03-29T11:06:28.164 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.185+0000 7f45f669d640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:28.164 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.185+0000 7f45f669d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:06:28.164 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.186+0000 7f45f669d640 10 monclient(hunting): picked mon.noname-b con 0x7f45f01457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:06:28.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.186+0000 7f45f669d640 10 monclient(hunting): picked mon.noname-c con 0x7f45f013f650 addr v1:172.21.15.137:6789/0 2024-03-29T11:06:28.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.188+0000 7f45effff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f45f01457f0 0x7f45f013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:51022/0 (socket says 172.21.15.71:51022) 2024-03-29T11:06:28.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.188+0000 7f45effff640 1 -- 172.21.15.71:0/2827900013 learned_addr learned my addr 172.21.15.71:0/2827900013 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:06:28.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.188+0000 7f45f669d640 10 monclient(hunting): picked mon.noname-a con 0x7f45f00a9d70 addr v1:172.21.15.71:6789/0 2024-03-29T11:06:28.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.188+0000 7f45f669d640 1 -- 172.21.15.71:0/2827900013 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45f00aaef0 con 0x7f45f00a9d70 2024-03-29T11:06:28.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.188+0000 7f45f669d640 1 -- 172.21.15.71:0/2827900013 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45f0145bd0 con 0x7f45f013f650 2024-03-29T11:06:28.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.188+0000 7f45f669d640 1 -- 172.21.15.71:0/2827900013 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45f0145eb0 con 0x7f45f01457f0 2024-03-29T11:06:28.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.188+0000 7f45f669d640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:28.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.189+0000 7f45f669d640 10 monclient(hunting): authenticate will time out at 1144.447159s 2024-03-29T11:06:28.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.190+0000 7f45eeffd640 1 -- 172.21.15.71:0/2827900013 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1534845226 0 0) 0x7f45f0145eb0 con 0x7f45f01457f0 2024-03-29T11:06:28.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.190+0000 7f45eeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:06:28.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.190+0000 7f45eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:28.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.190+0000 7f45eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:28.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.190+0000 7f45eeffd640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:06:28.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.190+0000 7f45eeffd640 1 -- 172.21.15.71:0/2827900013 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45dc003520 con 0x7f45f01457f0 2024-03-29T11:06:28.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.192+0000 7f45eeffd640 1 -- 172.21.15.71:0/2827900013 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 539204179 0 0) 0x7f45f00aaef0 con 0x7f45f00a9d70 2024-03-29T11:06:28.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.192+0000 7f45eeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:06:28.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.192+0000 7f45eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:28.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.192+0000 7f45eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:28.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.192+0000 7f45eeffd640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:06:28.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.192+0000 7f45eeffd640 1 -- 172.21.15.71:0/2827900013 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45f0145eb0 con 0x7f45f00a9d70 2024-03-29T11:06:28.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.192+0000 7f45eeffd640 1 -- 172.21.15.71:0/2827900013 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413548736 0 0) 0x7f45f0145bd0 con 0x7f45f013f650 2024-03-29T11:06:28.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.192+0000 7f45eeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:06:28.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.192+0000 7f45eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:28.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.192+0000 7f45eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:28.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.192+0000 7f45eeffd640 10 monclient(hunting): my global_id is 4411 2024-03-29T11:06:28.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.192+0000 7f45eeffd640 1 -- 172.21.15.71:0/2827900013 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45f00aaef0 con 0x7f45f013f650 2024-03-29T11:06:28.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.192+0000 7f45eeffd640 1 -- 172.21.15.71:0/2827900013 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3638338388 0 0) 0x7f45f0145eb0 con 0x7f45f00a9d70 2024-03-29T11:06:28.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.192+0000 7f45eeffd640 1 -- 172.21.15.71:0/2827900013 >> v1:172.21.15.71:6790/0 conn(0x7f45f01457f0 legacy=0x7f45f013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:28.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.192+0000 7f45eeffd640 1 -- 172.21.15.71:0/2827900013 >> v1:172.21.15.137:6789/0 conn(0x7f45f013f650 legacy=0x7f45f00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:28.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.192+0000 7f45eeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:06:28.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.192+0000 7f45eeffd640 1 monclient: found mon.noname-a 2024-03-29T11:06:28.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.192+0000 7f45eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:28.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.192+0000 7f45eeffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:06:28.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.192+0000 7f45eeffd640 1 -- 172.21.15.71:0/2827900013 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f0146340 con 0x7f45f00a9d70 2024-03-29T11:06:28.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.193+0000 7f45eeffd640 10 monclient: _finish_auth 0 2024-03-29T11:06:28.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.193+0000 7f45eeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:06:28.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.193+0000 7f45eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:28.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.193+0000 7f45eeffd640 1 -- 172.21.15.71:0/2827900013 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f45d8001670 con 0x7f45f00a9d70 2024-03-29T11:06:28.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.193+0000 7f45eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:28.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.193+0000 7f45eeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:06:28.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.193+0000 7f45eeffd640 10 monclient: dump: 2024-03-29T11:06:28.174 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:28.174 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:28.174 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:28.174 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:28.174 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:28.175 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:28.175 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:28.175 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:28.175 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:28.175 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:28.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.193+0000 7f45f669d640 5 monclient: authenticate success, global_id 4449 2024-03-29T11:06:28.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.193+0000 7f45f669d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:06:28.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.193+0000 7f45eeffd640 1 -- 172.21.15.71:0/2827900013 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45d8001890 con 0x7f45f00a9d70 2024-03-29T11:06:28.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.193+0000 7f45eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:28.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.193+0000 7f45eeffd640 1 -- 172.21.15.71:0/2827900013 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f45d8001c20 con 0x7f45f00a9d70 2024-03-29T11:06:28.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.193+0000 7f45eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:28.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.193+0000 7f45eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:06:28.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.193+0000 7f45eeffd640 10 monclient: dump: 2024-03-29T11:06:28.176 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:28.176 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:28.176 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:28.176 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:28.176 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:28.176 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:28.176 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:28.176 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:28.176 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:28.176 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:28.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.193+0000 7f45f669d640 10 monclient: get_monmap_and_config success 2024-03-29T11:06:28.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.193+0000 7f45f669d640 4 set_mon_vals no callback set 2024-03-29T11:06:28.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.194+0000 7f45f669d640 10 monclient: shutdown 2024-03-29T11:06:28.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.194+0000 7f45f669d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:28.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.194+0000 7f45f669d640 1 -- 172.21.15.71:0/2827900013 >> v1:172.21.15.71:6789/0 conn(0x7f45f00a9d70 legacy=0x7f45f014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:28.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.194+0000 7f45f669d640 1 -- 172.21.15.71:0/2827900013 shutdown_connections 2024-03-29T11:06:28.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.194+0000 7f45f669d640 1 -- 172.21.15.71:0/2827900013 >> 172.21.15.71:0/2827900013 conn(0x7f45f0066a00 msgr2=0x7f45f0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:28.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.194+0000 7f45f669d640 1 -- 172.21.15.71:0/2827900013 shutdown_connections 2024-03-29T11:06:28.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.194+0000 7f45f669d640 1 -- 172.21.15.71:0/2827900013 wait complete. 2024-03-29T11:06:28.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.195+0000 7f45f669d640 10 monclient: build_initial_monmap 2024-03-29T11:06:28.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.195+0000 7f45f669d640 10 monclient: monmap: 2024-03-29T11:06:28.182 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:06:28.182 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:06:28.182 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:06:28.182 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:06:28.182 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:28.182 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:28.182 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:06:28.182 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:06:28.182 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:06:28.182 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:28.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.195+0000 7f45f669d640 1 librados: starting msgr at 2024-03-29T11:06:28.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.195+0000 7f45f669d640 1 librados: starting objecter 2024-03-29T11:06:28.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.195+0000 7f45f669d640 1 Processor -- start 2024-03-29T11:06:28.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.196+0000 7f45f669d640 1 -- start start 2024-03-29T11:06:28.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.196+0000 7f45f669d640 1 librados: setting wanted keys 2024-03-29T11:06:28.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.196+0000 7f45f669d640 1 librados: calling monclient init 2024-03-29T11:06:28.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.196+0000 7f45f669d640 10 monclient: init 2024-03-29T11:06:28.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.197+0000 7f45f669d640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:28.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.197+0000 7f45f669d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:06:28.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.197+0000 7f45f669d640 10 monclient(hunting): picked mon.noname-c con 0x7f45f00a9d70 addr v1:172.21.15.71:6790/0 2024-03-29T11:06:28.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.197+0000 7f45effff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f45f00a9d70 0x7f45f00ab870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:52562/0 (socket says 172.21.15.71:52562) 2024-03-29T11:06:28.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.197+0000 7f45effff640 1 -- 172.21.15.71:0/2510336226 learned_addr learned my addr 172.21.15.71:0/2510336226 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:06:28.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.197+0000 7f45f669d640 10 monclient(hunting): picked mon.noname-b con 0x7f45f013f650 addr v1:172.21.15.137:6789/0 2024-03-29T11:06:28.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.198+0000 7f45f669d640 10 monclient(hunting): picked mon.noname-a con 0x7f45f01457f0 addr v1:172.21.15.71:6789/0 2024-03-29T11:06:28.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.198+0000 7f45f669d640 1 -- 172.21.15.71:0/2510336226 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45f00abf80 con 0x7f45f01457f0 2024-03-29T11:06:28.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.198+0000 7f45f669d640 1 -- 172.21.15.71:0/2510336226 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45f00ad8d0 con 0x7f45f013f650 2024-03-29T11:06:28.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.198+0000 7f45f669d640 1 -- 172.21.15.71:0/2510336226 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45f00adbb0 con 0x7f45f00a9d70 2024-03-29T11:06:28.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.198+0000 7f45f669d640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:28.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.198+0000 7f45f669d640 10 monclient(hunting): authenticate will time out at 1144.456412s 2024-03-29T11:06:28.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.199+0000 7f45ecff9640 10 client.?.objecter ms_handle_connect 0x7f45f00a9d70 2024-03-29T11:06:28.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.199+0000 7f45ecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:28.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.199+0000 7f45ecff9640 1 -- 172.21.15.71:0/2510336226 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3787705182 0 0) 0x7f45f00adbb0 con 0x7f45f00a9d70 2024-03-29T11:06:28.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.199+0000 7f45ecff9640 10 monclient(hunting): _negotiate 2024-03-29T11:06:28.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.199+0000 7f45ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:28.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.199+0000 7f45ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:28.185 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.199+0000 7f45ecff9640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:06:28.185 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.199+0000 7f45ecff9640 1 -- 172.21.15.71:0/2510336226 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45d0004eb0 con 0x7f45f00a9d70 2024-03-29T11:06:28.185 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.199+0000 7f45ecff9640 1 -- 172.21.15.71:0/2510336226 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3477511851 0 0) 0x7f45d0004eb0 con 0x7f45f00a9d70 2024-03-29T11:06:28.185 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.199+0000 7f45ecff9640 1 -- 172.21.15.71:0/2510336226 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45f00adbb0 con 0x7f45f00a9d70 2024-03-29T11:06:28.185 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.199+0000 7f45ecff9640 1 -- 172.21.15.71:0/2510336226 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f45e00018b0 con 0x7f45f00a9d70 2024-03-29T11:06:28.185 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.199+0000 7f45ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:28.185 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.199+0000 7f45ecff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:06:28.185 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.199+0000 7f45ecff9640 10 monclient(hunting): dump: 2024-03-29T11:06:28.185 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:28.185 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:28.185 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:28.185 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:28.186 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:28.186 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:28.186 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:28.186 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:28.186 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:28.186 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:28.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.199+0000 7f45ecff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:06:28.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.200+0000 7f45ecff9640 1 -- 172.21.15.71:0/2510336226 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4126530163 0 0) 0x7f45f00adbb0 con 0x7f45f00a9d70 2024-03-29T11:06:28.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.200+0000 7f45ecff9640 1 -- 172.21.15.71:0/2510336226 >> v1:172.21.15.137:6789/0 conn(0x7f45f013f650 legacy=0x7f45f0140950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:06:28.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.203+0000 7f45ecff9640 1 -- 172.21.15.71:0/2510336226 >> v1:172.21.15.71:6789/0 conn(0x7f45f01457f0 legacy=0x7f45f0141060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:28.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.203+0000 7f45ecff9640 10 monclient: _finish_hunting 0 2024-03-29T11:06:28.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.203+0000 7f45ecff9640 1 monclient: found mon.c 2024-03-29T11:06:28.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.203+0000 7f45ecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:28.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.203+0000 7f45ecff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:06:28.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.203+0000 7f45ecff9640 1 -- 172.21.15.71:0/2510336226 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f0138cb0 con 0x7f45f00a9d70 2024-03-29T11:06:28.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.203+0000 7f45ecff9640 10 monclient: _finish_auth 0 2024-03-29T11:06:28.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.203+0000 7f45ecff9640 10 monclient: _check_auth_tickets 2024-03-29T11:06:28.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.203+0000 7f45ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:28.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.203+0000 7f45ecff9640 10 client.?.objecter ms_handle_connect 0x7f45f01457f0 2024-03-29T11:06:28.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.203+0000 7f45ecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:28.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.206+0000 7f45f669d640 5 monclient: authenticate success, global_id 4484 2024-03-29T11:06:28.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.206+0000 7f45f669d640 10 monclient: _renew_subs 2024-03-29T11:06:28.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.206+0000 7f45f669d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:06:28.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.206+0000 7f45ecff9640 1 -- 172.21.15.71:0/2510336226 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45e00047e0 con 0x7f45f00a9d70 2024-03-29T11:06:28.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.206+0000 7f45f669d640 1 -- 172.21.15.71:0/2510336226 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45f01e2de0 con 0x7f45f00a9d70 2024-03-29T11:06:28.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.206+0000 7f45ecff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:28.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.206+0000 7f45ecff9640 1 -- 172.21.15.71:0/2510336226 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f45e00018b0 con 0x7f45f00a9d70 2024-03-29T11:06:28.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.206+0000 7f45ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:28.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.206+0000 7f45ecff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:06:28.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.206+0000 7f45ecff9640 10 monclient: dump: 2024-03-29T11:06:28.192 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:28.192 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:28.192 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:28.192 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:28.192 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:28.192 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:28.192 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:28.192 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:28.192 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:28.192 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:28.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.206+0000 7f45ee7fc640 4 set_mon_vals no callback set 2024-03-29T11:06:28.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.207+0000 7f45f669d640 10 client.4484.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:06:28.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.207+0000 7f45f669d640 10 monclient: _renew_subs 2024-03-29T11:06:28.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.207+0000 7f45f669d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:06:28.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.207+0000 7f45f669d640 1 -- 172.21.15.71:0/2510336226 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45f01e3150 con 0x7f45f00a9d70 2024-03-29T11:06:28.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.207+0000 7f45f669d640 1 librados: init done 2024-03-29T11:06:28.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.209+0000 7f45ecff9640 1 -- 172.21.15.71:0/2510336226 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f45e001a880 con 0x7f45f00a9d70 2024-03-29T11:06:28.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.209+0000 7f45f669d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:28.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.209+0000 7f45f669d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:28.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.209+0000 7f45f669d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:06:28.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.209+0000 7f45f669d640 1 -- 172.21.15.71:0/2510336226 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45f0075f10 con 0x7f45f00a9d70 2024-03-29T11:06:28.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.209+0000 7f45ecff9640 1 -- 172.21.15.71:0/2510336226 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f45e0084860 con 0x7f45f00a9d70 2024-03-29T11:06:28.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.209+0000 7f45ecff9640 10 client.4484.objecter ms_dispatch 0x7f45f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:06:28.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.209+0000 7f45ecff9640 3 client.4484.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:06:28.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.209+0000 7f45ecff9640 3 client.4484.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:06:28.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.209+0000 7f45ecff9640 20 client.4484.objecter dump_active .. 0 homeless 2024-03-29T11:06:28.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.209+0000 7f45ef7fe640 10 monclient: get_auth_request con 0x7f45d006dd40 auth_method 0 2024-03-29T11:06:28.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.210+0000 7f45ecff9640 10 client.4484.objecter ms_handle_connect 0x7f45d006dd40 2024-03-29T11:06:28.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.221+0000 7f45ecff9640 1 -- 172.21.15.71:0/2510336226 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f45e00501a0 con 0x7f45f00a9d70 2024-03-29T11:06:28.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.221+0000 7f45ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:28.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.221+0000 7f45ecff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:06:28.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.550+0000 7f45f669d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:28.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.550+0000 7f45f669d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:28.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.550+0000 7f45f669d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:06:28.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.550+0000 7f45f669d640 1 -- 172.21.15.71:0/2510336226 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45f00b3680 con 0x7f45f00a9d70 2024-03-29T11:06:28.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.552+0000 7f45ecff9640 1 -- 172.21.15.71:0/2510336226 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f45e00520d0 con 0x7f45f00a9d70 2024-03-29T11:06:28.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.552+0000 7f45ecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:28.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.552+0000 7f45ecff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:06:28.532 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:06:28.532 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:06:28.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.556+0000 7f45f669d640 10 librados: watch_flush enter 2024-03-29T11:06:28.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.556+0000 7f45f669d640 10 librados: watch_flush exit 2024-03-29T11:06:28.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.556+0000 7f45f669d640 20 client.4484.objecter shutdown clearing up homeless session... 2024-03-29T11:06:28.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.556+0000 7f45f669d640 10 client.4484.objecter successfully canceled tick 2024-03-29T11:06:28.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.557+0000 7f45f669d640 1 -- 172.21.15.71:0/2510336226 >> v1:172.21.15.137:6816/973332665 conn(0x7f45d006dd40 legacy=0x7f45d0070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:28.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.557+0000 7f45f669d640 10 monclient: shutdown 2024-03-29T11:06:28.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.557+0000 7f45f669d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:28.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.557+0000 7f45f669d640 1 -- 172.21.15.71:0/2510336226 >> v1:172.21.15.71:6790/0 conn(0x7f45f00a9d70 legacy=0x7f45f00ab870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:28.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.557+0000 7f45f669d640 1 -- 172.21.15.71:0/2510336226 shutdown_connections 2024-03-29T11:06:28.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.557+0000 7f45f669d640 1 -- 172.21.15.71:0/2510336226 >> 172.21.15.71:0/2510336226 conn(0x7f45f0066a00 msgr2=0x7f45f00b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:28.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.558+0000 7f45f669d640 1 -- 172.21.15.71:0/2510336226 shutdown_connections 2024-03-29T11:06:28.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.558+0000 7f45f669d640 1 -- 172.21.15.71:0/2510336226 wait complete. 2024-03-29T11:06:28.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:28.558+0000 7f45f669d640 1 librados: shutdown 2024-03-29T11:06:28.553 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:06:35.360 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:06:35.506 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.527+0000 7f7fcb4e4640 10 monclient: get_monmap_and_config 2024-03-29T11:06:35.506 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.527+0000 7f7fcb4e4640 10 monclient: build_initial_monmap 2024-03-29T11:06:35.506 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.527+0000 7f7fcb4e4640 10 monclient: monmap: 2024-03-29T11:06:35.506 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:06:35.506 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:35.506 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:06:35.527914+0000 2024-03-29T11:06:35.506 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:06:35.527914+0000 2024-03-29T11:06:35.506 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:35.507 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:35.507 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:06:35.507 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:06:35.507 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:06:35.507 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:35.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.527+0000 7f7fcb4e4640 1 Processor -- start 2024-03-29T11:06:35.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.528+0000 7f7fcb4e4640 1 -- start start 2024-03-29T11:06:35.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.528+0000 7f7fcb4e4640 10 monclient: init 2024-03-29T11:06:35.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.529+0000 7f7fcb4e4640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:35.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.529+0000 7f7fcb4e4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:06:35.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.529+0000 7f7fcb4e4640 10 monclient(hunting): picked mon.noname-a con 0x7f7fc406cac0 addr v1:172.21.15.71:6789/0 2024-03-29T11:06:35.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.529+0000 7f7fcb4e4640 10 monclient(hunting): picked mon.noname-b con 0x7f7fc406cea0 addr v1:172.21.15.71:6790/0 2024-03-29T11:06:35.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.529+0000 7f7fcb4e4640 10 monclient(hunting): picked mon.noname-c con 0x7f7fc4145e80 addr v1:172.21.15.137:6789/0 2024-03-29T11:06:35.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.529+0000 7f7fcb4e4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc40ac850 con 0x7f7fc406cac0 2024-03-29T11:06:35.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.529+0000 7f7fcb4e4640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc4070470 con 0x7f7fc4145e80 2024-03-29T11:06:35.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.529+0000 7f7fcb4e4640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc4072600 con 0x7f7fc406cea0 2024-03-29T11:06:35.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.529+0000 7f7fcb4e4640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:35.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.529+0000 7f7fcb4e4640 10 monclient(hunting): authenticate will time out at 1151.787225s 2024-03-29T11:06:35.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.534+0000 7f7fc9a81640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f7fc4145e80 0x7f7fc414db10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:54907/0 (socket says 172.21.15.71:54907) 2024-03-29T11:06:35.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.534+0000 7f7fc9a81640 1 -- 172.21.15.71:0/4196499255 learned_addr learned my addr 172.21.15.71:0/4196499255 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:06:35.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.534+0000 7f7fbbfff640 1 -- 172.21.15.71:0/4196499255 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2745611129 0 0) 0x7f7fc40ac850 con 0x7f7fc406cac0 2024-03-29T11:06:35.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.534+0000 7f7fbbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:06:35.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.534+0000 7f7fbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:35.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.534+0000 7f7fbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:35.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.534+0000 7f7fbbfff640 10 monclient(hunting): my global_id is 4455 2024-03-29T11:06:35.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.534+0000 7f7fbbfff640 1 -- 172.21.15.71:0/4196499255 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fb0003520 con 0x7f7fc406cac0 2024-03-29T11:06:35.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.534+0000 7f7fbbfff640 1 -- 172.21.15.71:0/4196499255 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356509156 0 0) 0x7f7fc4072600 con 0x7f7fc406cea0 2024-03-29T11:06:35.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.534+0000 7f7fbbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:06:35.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.534+0000 7f7fbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:35.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.534+0000 7f7fbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:35.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.534+0000 7f7fbbfff640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:06:35.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.534+0000 7f7fbbfff640 1 -- 172.21.15.71:0/4196499255 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fb0003840 con 0x7f7fc406cea0 2024-03-29T11:06:35.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.535+0000 7f7fbbfff640 1 -- 172.21.15.71:0/4196499255 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1051360257 0 0) 0x7f7fb0003520 con 0x7f7fc406cac0 2024-03-29T11:06:35.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.535+0000 7f7fbbfff640 1 -- 172.21.15.71:0/4196499255 >> v1:172.21.15.71:6790/0 conn(0x7f7fc406cea0 legacy=0x7f7fc4145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:35.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.535+0000 7f7fbbfff640 1 -- 172.21.15.71:0/4196499255 >> v1:172.21.15.137:6789/0 conn(0x7f7fc4145e80 legacy=0x7f7fc414db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:06:35.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.535+0000 7f7fbbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:06:35.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.535+0000 7f7fbbfff640 1 monclient: found mon.noname-a 2024-03-29T11:06:35.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.535+0000 7f7fbbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:35.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.535+0000 7f7fbbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:06:35.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.535+0000 7f7fbbfff640 1 -- 172.21.15.71:0/4196499255 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc41462d0 con 0x7f7fc406cac0 2024-03-29T11:06:35.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.535+0000 7f7fbbfff640 10 monclient: _finish_auth 0 2024-03-29T11:06:35.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.535+0000 7f7fbbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:06:35.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.535+0000 7f7fbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:35.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.535+0000 7f7fcb4e4640 5 monclient: authenticate success, global_id 4455 2024-03-29T11:06:35.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.535+0000 7f7fcb4e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:06:35.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.535+0000 7f7fbbfff640 1 -- 172.21.15.71:0/4196499255 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f7fb4001670 con 0x7f7fc406cac0 2024-03-29T11:06:35.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.535+0000 7f7fbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:35.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.535+0000 7f7fbbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:06:35.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.535+0000 7f7fbbfff640 10 monclient: dump: 2024-03-29T11:06:35.519 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:35.519 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:35.520 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:35.520 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:35.520 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:35.520 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:35.520 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:35.520 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:35.520 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:35.520 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:35.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.535+0000 7f7fcb4e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:06:35.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.535+0000 7f7fbbfff640 1 -- 172.21.15.71:0/4196499255 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fb4001850 con 0x7f7fc406cac0 2024-03-29T11:06:35.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.535+0000 7f7fbbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:35.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.535+0000 7f7fbbfff640 1 -- 172.21.15.71:0/4196499255 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f7fb4001c60 con 0x7f7fc406cac0 2024-03-29T11:06:35.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.535+0000 7f7fbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:35.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.535+0000 7f7fbbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:06:35.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.535+0000 7f7fbbfff640 10 monclient: dump: 2024-03-29T11:06:35.521 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:35.521 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:35.521 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:35.521 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:35.521 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:35.521 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:35.521 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:35.521 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:35.521 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:35.521 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:35.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.535+0000 7f7fcb4e4640 10 monclient: get_monmap_and_config success 2024-03-29T11:06:35.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.535+0000 7f7fcb4e4640 4 set_mon_vals no callback set 2024-03-29T11:06:35.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.536+0000 7f7fcb4e4640 10 monclient: shutdown 2024-03-29T11:06:35.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.536+0000 7f7fcb4e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:35.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.536+0000 7f7fcb4e4640 1 -- 172.21.15.71:0/4196499255 >> v1:172.21.15.71:6789/0 conn(0x7f7fc406cac0 legacy=0x7f7fc4071ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:35.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.538+0000 7f7fcb4e4640 1 -- 172.21.15.71:0/4196499255 shutdown_connections 2024-03-29T11:06:35.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.538+0000 7f7fcb4e4640 1 -- 172.21.15.71:0/4196499255 >> 172.21.15.71:0/4196499255 conn(0x7f7fc40675b0 msgr2=0x7f7fc40b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:35.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.541+0000 7f7fcb4e4640 1 -- 172.21.15.71:0/4196499255 shutdown_connections 2024-03-29T11:06:35.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.541+0000 7f7fcb4e4640 1 -- 172.21.15.71:0/4196499255 wait complete. 2024-03-29T11:06:35.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.542+0000 7f7fcb4e4640 10 monclient: build_initial_monmap 2024-03-29T11:06:35.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.542+0000 7f7fcb4e4640 10 monclient: monmap: 2024-03-29T11:06:35.523 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:06:35.523 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:06:35.523 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:06:35.523 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:06:35.524 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:35.524 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:35.524 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:06:35.524 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:06:35.524 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:06:35.524 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:35.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.542+0000 7f7fcb4e4640 1 librados: starting msgr at 2024-03-29T11:06:35.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.542+0000 7f7fcb4e4640 1 librados: starting objecter 2024-03-29T11:06:35.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.543+0000 7f7fcb4e4640 1 Processor -- start 2024-03-29T11:06:35.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.544+0000 7f7fcb4e4640 1 -- start start 2024-03-29T11:06:35.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.544+0000 7f7fcb4e4640 1 librados: setting wanted keys 2024-03-29T11:06:35.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.544+0000 7f7fcb4e4640 1 librados: calling monclient init 2024-03-29T11:06:35.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.544+0000 7f7fcb4e4640 10 monclient: init 2024-03-29T11:06:35.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.545+0000 7f7fcb4e4640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:35.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.545+0000 7f7fcb4e4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:06:35.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.545+0000 7f7fcb4e4640 10 monclient(hunting): picked mon.noname-b con 0x7f7fc406cac0 addr v1:172.21.15.137:6789/0 2024-03-29T11:06:35.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.549+0000 7f7fcb4e4640 10 monclient(hunting): picked mon.noname-a con 0x7f7fc406cea0 addr v1:172.21.15.71:6789/0 2024-03-29T11:06:35.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.549+0000 7f7fc8a7f640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f7fc406cea0 0x7f7fc4140050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:58129/0 (socket says 172.21.15.71:58129) 2024-03-29T11:06:35.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.549+0000 7f7fc8a7f640 1 -- 172.21.15.71:0/2489375073 learned_addr learned my addr 172.21.15.71:0/2489375073 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:06:35.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.549+0000 7f7fcb4e4640 10 monclient(hunting): picked mon.noname-c con 0x7f7fc4145e80 addr v1:172.21.15.71:6790/0 2024-03-29T11:06:35.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.549+0000 7f7fcb4e4640 1 -- 172.21.15.71:0/2489375073 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc40ad1a0 con 0x7f7fc406cea0 2024-03-29T11:06:35.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.549+0000 7f7fcb4e4640 1 -- 172.21.15.71:0/2489375073 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc413f9a0 con 0x7f7fc406cac0 2024-03-29T11:06:35.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.549+0000 7f7fcb4e4640 1 -- 172.21.15.71:0/2489375073 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc4141f20 con 0x7f7fc4145e80 2024-03-29T11:06:35.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fcb4e4640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:35.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fcb4e4640 10 monclient(hunting): authenticate will time out at 1151.808263s 2024-03-29T11:06:35.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 10 client.?.objecter ms_handle_connect 0x7f7fc406cea0 2024-03-29T11:06:35.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:35.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 10 client.?.objecter ms_handle_connect 0x7f7fc4145e80 2024-03-29T11:06:35.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:35.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 1 -- 172.21.15.71:0/2489375073 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364794956 0 0) 0x7f7fc40ad1a0 con 0x7f7fc406cea0 2024-03-29T11:06:35.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:06:35.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:35.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:35.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 10 monclient(hunting): my global_id is 4458 2024-03-29T11:06:35.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 1 -- 172.21.15.71:0/2489375073 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fa8004eb0 con 0x7f7fc406cea0 2024-03-29T11:06:35.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 1 -- 172.21.15.71:0/2489375073 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2970883767 0 0) 0x7f7fc4141f20 con 0x7f7fc4145e80 2024-03-29T11:06:35.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:06:35.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:35.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:35.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:06:35.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 1 -- 172.21.15.71:0/2489375073 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc40ad1a0 con 0x7f7fc4145e80 2024-03-29T11:06:35.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 1 -- 172.21.15.71:0/2489375073 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1227238661 0 0) 0x7f7fc40ad1a0 con 0x7f7fc4145e80 2024-03-29T11:06:35.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 1 -- 172.21.15.71:0/2489375073 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fc4141f20 con 0x7f7fc4145e80 2024-03-29T11:06:35.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 1 -- 172.21.15.71:0/2489375073 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4243999303 0 0) 0x7f7fa8004eb0 con 0x7f7fc406cea0 2024-03-29T11:06:35.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 1 -- 172.21.15.71:0/2489375073 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fc40ad1a0 con 0x7f7fc406cea0 2024-03-29T11:06:35.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 1 -- 172.21.15.71:0/2489375073 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f7fbc0019c0 con 0x7f7fc4145e80 2024-03-29T11:06:35.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:35.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:06:35.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 10 monclient(hunting): dump: 2024-03-29T11:06:35.535 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:35.535 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:35.535 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:35.535 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:35.535 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:35.536 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:35.536 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:35.536 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:35.536 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:35.536 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:06:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 1 -- 172.21.15.71:0/2489375073 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f7fc0001970 con 0x7f7fc406cea0 2024-03-29T11:06:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:06:35.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.550+0000 7f7fb9ffb640 10 monclient(hunting): dump: 2024-03-29T11:06:35.536 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:35.536 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:35.536 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:35.536 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:35.537 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:35.537 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:35.537 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:35.537 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:35.537 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:35.537 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:35.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.551+0000 7f7fb9ffb640 1 -- 172.21.15.71:0/2489375073 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2925437384 0 0) 0x7f7fc4141f20 con 0x7f7fc4145e80 2024-03-29T11:06:35.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.551+0000 7f7fb9ffb640 1 -- 172.21.15.71:0/2489375073 >> v1:172.21.15.137:6789/0 conn(0x7f7fc406cac0 legacy=0x7f7fc413f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:35.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.551+0000 7f7fb9ffb640 1 -- 172.21.15.71:0/2489375073 >> v1:172.21.15.71:6789/0 conn(0x7f7fc406cea0 legacy=0x7f7fc4140050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:35.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.551+0000 7f7fb9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:06:35.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.551+0000 7f7fb9ffb640 1 monclient: found mon.c 2024-03-29T11:06:35.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.551+0000 7f7fb9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:35.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.551+0000 7f7fb9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:06:35.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.551+0000 7f7fb9ffb640 1 -- 172.21.15.71:0/2489375073 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc41428a0 con 0x7f7fc4145e80 2024-03-29T11:06:35.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.551+0000 7f7fb9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:06:35.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.551+0000 7f7fb9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:06:35.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.551+0000 7f7fb9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:35.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.551+0000 7f7fcb4e4640 5 monclient: authenticate success, global_id 4490 2024-03-29T11:06:35.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.551+0000 7f7fcb4e4640 10 monclient: _renew_subs 2024-03-29T11:06:35.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.551+0000 7f7fcb4e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:06:35.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.551+0000 7f7fcb4e4640 1 -- 172.21.15.71:0/2489375073 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fc4140e70 con 0x7f7fc4145e80 2024-03-29T11:06:35.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.551+0000 7f7fcb4e4640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:06:35.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.551+0000 7f7fcb4e4640 10 monclient: _renew_subs 2024-03-29T11:06:35.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.551+0000 7f7fcb4e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:06:35.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.551+0000 7f7fcb4e4640 1 -- 172.21.15.71:0/2489375073 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc4141210 con 0x7f7fc4145e80 2024-03-29T11:06:35.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.551+0000 7f7fcb4e4640 1 librados: init done 2024-03-29T11:06:35.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.552+0000 7f7fb9ffb640 1 -- 172.21.15.71:0/2489375073 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fbc001b20 con 0x7f7fc4145e80 2024-03-29T11:06:35.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.552+0000 7f7fb9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:35.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.552+0000 7f7fb9ffb640 1 -- 172.21.15.71:0/2489375073 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f7fbc004b60 con 0x7f7fc4145e80 2024-03-29T11:06:35.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.552+0000 7f7fb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:35.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.552+0000 7f7fb9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:06:35.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.552+0000 7f7fb9ffb640 10 monclient: dump: 2024-03-29T11:06:35.539 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:35.539 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:35.539 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:35.539 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:35.539 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:35.539 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:35.540 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:35.540 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:35.540 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:35.540 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:35.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.552+0000 7f7fbb7fe640 4 set_mon_vals no callback set 2024-03-29T11:06:35.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.555+0000 7f7fb9ffb640 1 -- 172.21.15.71:0/2489375073 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f7fbc004cc0 con 0x7f7fc4145e80 2024-03-29T11:06:35.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.555+0000 7f7fb9ffb640 1 -- 172.21.15.71:0/2489375073 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f7fbc084b80 con 0x7f7fc4145e80 2024-03-29T11:06:35.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.556+0000 7f7fb9ffb640 10 client.4490.objecter ms_dispatch 0x7f7fc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:06:35.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.556+0000 7f7fb9ffb640 3 client.4490.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:06:35.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.556+0000 7f7fb9ffb640 3 client.4490.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:06:35.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.556+0000 7f7fb9ffb640 20 client.4490.objecter dump_active .. 0 homeless 2024-03-29T11:06:35.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.557+0000 7f7fcb4e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:35.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.557+0000 7f7fcb4e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:35.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.557+0000 7f7fcb4e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:06:35.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.557+0000 7f7fcb4e4640 1 -- 172.21.15.71:0/2489375073 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fc40adc40 con 0x7f7fc4145e80 2024-03-29T11:06:35.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.557+0000 7f7fc9280640 10 monclient: get_auth_request con 0x7f7fa806ded0 auth_method 0 2024-03-29T11:06:35.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.558+0000 7f7fb9ffb640 10 client.4490.objecter ms_handle_connect 0x7f7fa806ded0 2024-03-29T11:06:35.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.566+0000 7f7fb9ffb640 1 -- 172.21.15.71:0/2489375073 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7fbc050290 con 0x7f7fc4145e80 2024-03-29T11:06:35.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.566+0000 7f7fb9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:35.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.566+0000 7f7fb9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:06:35.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.888+0000 7f7fcb4e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:35.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.889+0000 7f7fcb4e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:35.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.889+0000 7f7fcb4e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:06:35.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.889+0000 7f7fcb4e4640 1 -- 172.21.15.71:0/2489375073 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fc41420b0 con 0x7f7fc4145e80 2024-03-29T11:06:35.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.890+0000 7f7fb9ffb640 1 -- 172.21.15.71:0/2489375073 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f7fbc052390 con 0x7f7fc4145e80 2024-03-29T11:06:35.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.890+0000 7f7fb9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:35.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.890+0000 7f7fb9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:06:35.869 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:06:35.870 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:06:35.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.894+0000 7f7fcb4e4640 10 librados: watch_flush enter 2024-03-29T11:06:35.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.894+0000 7f7fcb4e4640 10 librados: watch_flush exit 2024-03-29T11:06:35.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.894+0000 7f7fcb4e4640 20 client.4490.objecter shutdown clearing up homeless session... 2024-03-29T11:06:35.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.894+0000 7f7fcb4e4640 10 client.4490.objecter successfully canceled tick 2024-03-29T11:06:35.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.894+0000 7f7fcb4e4640 1 -- 172.21.15.71:0/2489375073 >> v1:172.21.15.137:6816/973332665 conn(0x7f7fa806ded0 legacy=0x7f7fa8070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:35.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.894+0000 7f7fcb4e4640 10 monclient: shutdown 2024-03-29T11:06:35.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.894+0000 7f7fcb4e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:35.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.894+0000 7f7fcb4e4640 1 -- 172.21.15.71:0/2489375073 >> v1:172.21.15.71:6790/0 conn(0x7f7fc4145e80 legacy=0x7f7fc4140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:35.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.895+0000 7f7fcb4e4640 1 -- 172.21.15.71:0/2489375073 shutdown_connections 2024-03-29T11:06:35.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.895+0000 7f7fcb4e4640 1 -- 172.21.15.71:0/2489375073 >> 172.21.15.71:0/2489375073 conn(0x7f7fc40675b0 msgr2=0x7f7fc40b2500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:35.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.896+0000 7f7fcb4e4640 1 -- 172.21.15.71:0/2489375073 shutdown_connections 2024-03-29T11:06:35.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.896+0000 7f7fcb4e4640 1 -- 172.21.15.71:0/2489375073 wait complete. 2024-03-29T11:06:35.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:35.896+0000 7f7fcb4e4640 1 librados: shutdown 2024-03-29T11:06:35.893 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:06:42.701 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:06:42.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.868+0000 7fe4be13a640 10 monclient: get_monmap_and_config 2024-03-29T11:06:42.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.868+0000 7fe4be13a640 10 monclient: build_initial_monmap 2024-03-29T11:06:42.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.868+0000 7fe4be13a640 10 monclient: monmap: 2024-03-29T11:06:42.849 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:06:42.849 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:42.849 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:06:42.870683+0000 2024-03-29T11:06:42.849 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:06:42.870683+0000 2024-03-29T11:06:42.849 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:42.849 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:42.849 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:06:42.849 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:06:42.849 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:06:42.850 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:42.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.870+0000 7fe4be13a640 1 Processor -- start 2024-03-29T11:06:42.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.870+0000 7fe4be13a640 1 -- start start 2024-03-29T11:06:42.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.870+0000 7fe4be13a640 10 monclient: init 2024-03-29T11:06:42.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.871+0000 7fe4be13a640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:42.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.871+0000 7fe4be13a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:06:42.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.871+0000 7fe4be13a640 10 monclient(hunting): picked mon.noname-a con 0x7fe4b80b0860 addr v1:172.21.15.71:6789/0 2024-03-29T11:06:42.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.871+0000 7fe4be13a640 10 monclient(hunting): picked mon.noname-c con 0x7fe4b813ef30 addr v1:172.21.15.137:6789/0 2024-03-29T11:06:42.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.871+0000 7fe4be13a640 10 monclient(hunting): picked mon.noname-b con 0x7fe4b80af8e0 addr v1:172.21.15.71:6790/0 2024-03-29T11:06:42.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.871+0000 7fe4be13a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b80b0c40 con 0x7fe4b80b0860 2024-03-29T11:06:42.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.871+0000 7fe4b7fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fe4b80af8e0 0x7fe4b8149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:38202/0 (socket says 172.21.15.71:38202) 2024-03-29T11:06:42.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.871+0000 7fe4b7fff640 1 -- 172.21.15.71:0/2398000404 learned_addr learned my addr 172.21.15.71:0/2398000404 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:06:42.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.871+0000 7fe4be13a640 1 -- 172.21.15.71:0/2398000404 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b80aed10 con 0x7fe4b813ef30 2024-03-29T11:06:42.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.872+0000 7fe4be13a640 1 -- 172.21.15.71:0/2398000404 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b80afd50 con 0x7fe4b80af8e0 2024-03-29T11:06:42.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.872+0000 7fe4be13a640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:42.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.872+0000 7fe4be13a640 10 monclient(hunting): authenticate will time out at 1159.131322s 2024-03-29T11:06:42.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.873+0000 7fe4b67fc640 1 -- 172.21.15.71:0/2398000404 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1386047488 0 0) 0x7fe4b80afd50 con 0x7fe4b80af8e0 2024-03-29T11:06:42.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.873+0000 7fe4b67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:06:42.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.873+0000 7fe4b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:42.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.874+0000 7fe4b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:42.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.874+0000 7fe4b67fc640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:06:42.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.874+0000 7fe4b67fc640 1 -- 172.21.15.71:0/2398000404 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4a0003520 con 0x7fe4b80af8e0 2024-03-29T11:06:42.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.874+0000 7fe4b67fc640 1 -- 172.21.15.71:0/2398000404 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2969293780 0 0) 0x7fe4a0003520 con 0x7fe4b80af8e0 2024-03-29T11:06:42.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.874+0000 7fe4b67fc640 1 -- 172.21.15.71:0/2398000404 >> v1:172.21.15.137:6789/0 conn(0x7fe4b813ef30 legacy=0x7fe4b80af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:06:42.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.874+0000 7fe4b67fc640 1 -- 172.21.15.71:0/2398000404 >> v1:172.21.15.71:6789/0 conn(0x7fe4b80b0860 legacy=0x7fe4b813e820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:06:42.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.876+0000 7fe4b67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:06:42.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.876+0000 7fe4b67fc640 1 monclient: found mon.noname-b 2024-03-29T11:06:42.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.876+0000 7fe4b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:42.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.876+0000 7fe4b67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:06:42.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.876+0000 7fe4b67fc640 1 -- 172.21.15.71:0/2398000404 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b80ab1b0 con 0x7fe4b80af8e0 2024-03-29T11:06:42.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.876+0000 7fe4b67fc640 10 monclient: _finish_auth 0 2024-03-29T11:06:42.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.876+0000 7fe4b67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:06:42.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.876+0000 7fe4b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:42.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.876+0000 7fe4b67fc640 1 -- 172.21.15.71:0/2398000404 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fe4b0001a10 con 0x7fe4b80af8e0 2024-03-29T11:06:42.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.877+0000 7fe4be13a640 5 monclient: authenticate success, global_id 4493 2024-03-29T11:06:42.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.877+0000 7fe4be13a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:06:42.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.877+0000 7fe4b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:42.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.877+0000 7fe4b67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:06:42.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.877+0000 7fe4b67fc640 10 monclient: dump: 2024-03-29T11:06:42.858 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:42.858 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:42.858 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:42.858 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:42.858 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:42.859 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:42.859 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:42.859 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:42.859 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:42.859 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:42.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.877+0000 7fe4be13a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:06:42.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.877+0000 7fe4b67fc640 1 -- 172.21.15.71:0/2398000404 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4b0001ca0 con 0x7fe4b80af8e0 2024-03-29T11:06:42.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.877+0000 7fe4b67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:42.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.877+0000 7fe4be13a640 10 monclient: get_monmap_and_config success 2024-03-29T11:06:42.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.877+0000 7fe4be13a640 4 set_mon_vals no callback set 2024-03-29T11:06:42.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.878+0000 7fe4b67fc640 1 -- 172.21.15.71:0/2398000404 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fe4b0002030 con 0x7fe4b80af8e0 2024-03-29T11:06:42.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.878+0000 7fe4b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:42.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.878+0000 7fe4b67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:06:42.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.878+0000 7fe4b67fc640 10 monclient: dump: 2024-03-29T11:06:42.861 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:42.861 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:42.861 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:42.861 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:42.861 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:42.861 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:42.861 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:42.861 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:42.861 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:42.861 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:42.861 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.878+0000 7fe4be13a640 10 monclient: shutdown 2024-03-29T11:06:42.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.878+0000 7fe4be13a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:42.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.878+0000 7fe4be13a640 1 -- 172.21.15.71:0/2398000404 >> v1:172.21.15.71:6790/0 conn(0x7fe4b80af8e0 legacy=0x7fe4b8149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:42.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.880+0000 7fe4be13a640 1 -- 172.21.15.71:0/2398000404 shutdown_connections 2024-03-29T11:06:42.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.880+0000 7fe4be13a640 1 -- 172.21.15.71:0/2398000404 >> 172.21.15.71:0/2398000404 conn(0x7fe4b806b660 msgr2=0x7fe4b806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:42.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.880+0000 7fe4be13a640 1 -- 172.21.15.71:0/2398000404 shutdown_connections 2024-03-29T11:06:42.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.881+0000 7fe4be13a640 1 -- 172.21.15.71:0/2398000404 wait complete. 2024-03-29T11:06:42.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.882+0000 7fe4be13a640 10 monclient: build_initial_monmap 2024-03-29T11:06:42.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.882+0000 7fe4be13a640 10 monclient: monmap: 2024-03-29T11:06:42.863 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:06:42.863 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:06:42.863 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:06:42.863 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:06:42.863 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:42.863 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:42.863 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:06:42.863 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:06:42.863 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:06:42.864 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:42.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.882+0000 7fe4be13a640 1 librados: starting msgr at 2024-03-29T11:06:42.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.882+0000 7fe4be13a640 1 librados: starting objecter 2024-03-29T11:06:42.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.883+0000 7fe4be13a640 1 Processor -- start 2024-03-29T11:06:42.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.883+0000 7fe4be13a640 1 -- start start 2024-03-29T11:06:42.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.883+0000 7fe4be13a640 1 librados: setting wanted keys 2024-03-29T11:06:42.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.883+0000 7fe4be13a640 1 librados: calling monclient init 2024-03-29T11:06:42.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.883+0000 7fe4be13a640 10 monclient: init 2024-03-29T11:06:42.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.884+0000 7fe4be13a640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:42.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.884+0000 7fe4be13a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:06:42.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.884+0000 7fe4be13a640 10 monclient(hunting): picked mon.noname-a con 0x7fe4b80af8e0 addr v1:172.21.15.71:6789/0 2024-03-29T11:06:42.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.884+0000 7fe4be13a640 10 monclient(hunting): picked mon.noname-b con 0x7fe4b80b0860 addr v1:172.21.15.137:6789/0 2024-03-29T11:06:42.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.884+0000 7fe4be13a640 10 monclient(hunting): picked mon.noname-c con 0x7fe4b813ef30 addr v1:172.21.15.71:6790/0 2024-03-29T11:06:42.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.884+0000 7fe4be13a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b81dfa40 con 0x7fe4b80af8e0 2024-03-29T11:06:42.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.884+0000 7fe4be13a640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b80b2600 con 0x7fe4b80b0860 2024-03-29T11:06:42.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.884+0000 7fe4be13a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b80b2790 con 0x7fe4b813ef30 2024-03-29T11:06:42.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.884+0000 7fe4be13a640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:42.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.884+0000 7fe4be13a640 10 monclient(hunting): authenticate will time out at 1159.143385s 2024-03-29T11:06:42.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.884+0000 7fe4b7fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fe4b813ef30 0x7fe4b80b3280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:39612/0 (socket says 172.21.15.71:39612) 2024-03-29T11:06:42.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.884+0000 7fe4b7fff640 1 -- 172.21.15.71:0/3385682570 learned_addr learned my addr 172.21.15.71:0/3385682570 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:06:42.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.885+0000 7fe49ffff640 10 client.?.objecter ms_handle_connect 0x7fe4b813ef30 2024-03-29T11:06:42.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.885+0000 7fe49ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:42.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.885+0000 7fe49ffff640 1 -- 172.21.15.71:0/3385682570 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675786627 0 0) 0x7fe4b80b2790 con 0x7fe4b813ef30 2024-03-29T11:06:42.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.885+0000 7fe49ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:06:42.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.885+0000 7fe49ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:42.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.885+0000 7fe49ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:42.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.885+0000 7fe49ffff640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:06:42.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.885+0000 7fe49ffff640 1 -- 172.21.15.71:0/3385682570 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4a4004eb0 con 0x7fe4b813ef30 2024-03-29T11:06:42.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.885+0000 7fe49ffff640 1 -- 172.21.15.71:0/3385682570 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 519939363 0 0) 0x7fe4a4004eb0 con 0x7fe4b813ef30 2024-03-29T11:06:42.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.886+0000 7fe49ffff640 1 -- 172.21.15.71:0/3385682570 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4b80b2790 con 0x7fe4b813ef30 2024-03-29T11:06:42.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.887+0000 7fe49ffff640 1 -- 172.21.15.71:0/3385682570 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fe4b0001990 con 0x7fe4b813ef30 2024-03-29T11:06:42.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.887+0000 7fe49ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:42.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.887+0000 7fe49ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:06:42.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.887+0000 7fe49ffff640 10 monclient(hunting): dump: 2024-03-29T11:06:42.872 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:42.872 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:42.872 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:42.872 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:42.872 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:42.872 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:42.872 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:42.872 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:42.872 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:42.872 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:42.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.887+0000 7fe49ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:06:42.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.889+0000 7fe49ffff640 1 -- 172.21.15.71:0/3385682570 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2139488114 0 0) 0x7fe4b80b2790 con 0x7fe4b813ef30 2024-03-29T11:06:42.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.889+0000 7fe49ffff640 1 -- 172.21.15.71:0/3385682570 >> v1:172.21.15.137:6789/0 conn(0x7fe4b80b0860 legacy=0x7fe4b80b2b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:06:42.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.889+0000 7fe49ffff640 1 -- 172.21.15.71:0/3385682570 >> v1:172.21.15.71:6789/0 conn(0x7fe4b80af8e0 legacy=0x7fe4b81348a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:42.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.889+0000 7fe49ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:06:42.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.889+0000 7fe49ffff640 1 monclient: found mon.c 2024-03-29T11:06:42.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.889+0000 7fe49ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:42.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.889+0000 7fe49ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:06:42.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.889+0000 7fe49ffff640 1 -- 172.21.15.71:0/3385682570 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b80b2920 con 0x7fe4b813ef30 2024-03-29T11:06:42.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.889+0000 7fe49ffff640 10 monclient: _finish_auth 0 2024-03-29T11:06:42.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.889+0000 7fe49ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:06:42.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.889+0000 7fe49ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:42.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.889+0000 7fe49ffff640 10 client.?.objecter ms_handle_connect 0x7fe4b80af8e0 2024-03-29T11:06:42.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.889+0000 7fe49ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:42.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.889+0000 7fe4be13a640 5 monclient: authenticate success, global_id 4496 2024-03-29T11:06:42.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.889+0000 7fe4be13a640 10 monclient: _renew_subs 2024-03-29T11:06:42.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.889+0000 7fe4be13a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:06:42.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.889+0000 7fe4be13a640 1 -- 172.21.15.71:0/3385682570 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b8137470 con 0x7fe4b813ef30 2024-03-29T11:06:42.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.889+0000 7fe4be13a640 10 client.4496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:06:42.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.889+0000 7fe4be13a640 10 monclient: _renew_subs 2024-03-29T11:06:42.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.889+0000 7fe4be13a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:06:42.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.889+0000 7fe4be13a640 1 -- 172.21.15.71:0/3385682570 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4b8135190 con 0x7fe4b813ef30 2024-03-29T11:06:42.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.890+0000 7fe4be13a640 1 librados: init done 2024-03-29T11:06:42.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.891+0000 7fe49ffff640 1 -- 172.21.15.71:0/3385682570 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4b0004f40 con 0x7fe4b813ef30 2024-03-29T11:06:42.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.891+0000 7fe49ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:42.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.891+0000 7fe49ffff640 1 -- 172.21.15.71:0/3385682570 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fe4b00050b0 con 0x7fe4b813ef30 2024-03-29T11:06:42.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.891+0000 7fe49ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:42.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.892+0000 7fe49ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:06:42.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.892+0000 7fe49ffff640 10 monclient: dump: 2024-03-29T11:06:42.876 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:42.876 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:42.876 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:42.876 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:42.876 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:42.876 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:42.876 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:42.876 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:42.876 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:42.877 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:42.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.893+0000 7fe49ffff640 1 -- 172.21.15.71:0/3385682570 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7fe4b001b080 con 0x7fe4b813ef30 2024-03-29T11:06:42.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.893+0000 7fe4be13a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:42.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.893+0000 7fe4be13a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:42.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.893+0000 7fe4be13a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:06:42.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.893+0000 7fe4be13a640 1 -- 172.21.15.71:0/3385682570 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4b8137810 con 0x7fe4b813ef30 2024-03-29T11:06:42.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.893+0000 7fe4b5ffb640 4 set_mon_vals no callback set 2024-03-29T11:06:42.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.894+0000 7fe49ffff640 1 -- 172.21.15.71:0/3385682570 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7fe4b0084ee0 con 0x7fe4b813ef30 2024-03-29T11:06:42.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.894+0000 7fe49ffff640 10 client.4496.objecter ms_dispatch 0x7fe4b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:06:42.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.894+0000 7fe49ffff640 3 client.4496.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:06:42.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.894+0000 7fe49ffff640 3 client.4496.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:06:42.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.894+0000 7fe49ffff640 20 client.4496.objecter dump_active .. 0 homeless 2024-03-29T11:06:42.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.895+0000 7fe4b77fe640 10 monclient: get_auth_request con 0x7fe4a406dba0 auth_method 0 2024-03-29T11:06:42.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.895+0000 7fe49ffff640 10 client.4496.objecter ms_handle_connect 0x7fe4a406dba0 2024-03-29T11:06:42.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.903+0000 7fe49ffff640 1 -- 172.21.15.71:0/3385682570 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe4b00507c0 con 0x7fe4b813ef30 2024-03-29T11:06:42.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.903+0000 7fe49ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:42.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:42.903+0000 7fe49ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:06:43.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:43.224+0000 7fe4be13a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:43.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:43.224+0000 7fe4be13a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:43.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:43.224+0000 7fe4be13a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:06:43.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:43.224+0000 7fe4be13a640 1 -- 172.21.15.71:0/3385682570 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4b81dfbd0 con 0x7fe4b813ef30 2024-03-29T11:06:43.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:43.225+0000 7fe49ffff640 1 -- 172.21.15.71:0/3385682570 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7fe4b00526f0 con 0x7fe4b813ef30 2024-03-29T11:06:43.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:43.225+0000 7fe49ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:43.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:43.225+0000 7fe49ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:06:43.205 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:06:43.205 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:06:43.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:43.228+0000 7fe4be13a640 10 librados: watch_flush enter 2024-03-29T11:06:43.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:43.228+0000 7fe4be13a640 10 librados: watch_flush exit 2024-03-29T11:06:43.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:43.228+0000 7fe4be13a640 20 client.4496.objecter shutdown clearing up homeless session... 2024-03-29T11:06:43.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:43.228+0000 7fe4be13a640 10 client.4496.objecter successfully canceled tick 2024-03-29T11:06:43.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:43.228+0000 7fe4be13a640 1 -- 172.21.15.71:0/3385682570 >> v1:172.21.15.137:6816/973332665 conn(0x7fe4a406dba0 legacy=0x7fe4a4070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:43.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:43.228+0000 7fe4be13a640 10 monclient: shutdown 2024-03-29T11:06:43.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:43.228+0000 7fe4be13a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:43.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:43.228+0000 7fe4be13a640 1 -- 172.21.15.71:0/3385682570 >> v1:172.21.15.71:6790/0 conn(0x7fe4b813ef30 legacy=0x7fe4b80b3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:43.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:43.230+0000 7fe4be13a640 1 -- 172.21.15.71:0/3385682570 shutdown_connections 2024-03-29T11:06:43.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:43.230+0000 7fe4be13a640 1 -- 172.21.15.71:0/3385682570 >> 172.21.15.71:0/3385682570 conn(0x7fe4b806b660 msgr2=0x7fe4b813d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:43.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:43.230+0000 7fe4be13a640 1 -- 172.21.15.71:0/3385682570 shutdown_connections 2024-03-29T11:06:43.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:43.231+0000 7fe4be13a640 1 -- 172.21.15.71:0/3385682570 wait complete. 2024-03-29T11:06:43.211 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:43.231+0000 7fe4be13a640 1 librados: shutdown 2024-03-29T11:06:43.229 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:06:50.036 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:06:50.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.201+0000 7ffaf3c9a640 10 monclient: get_monmap_and_config 2024-03-29T11:06:50.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.201+0000 7ffaf3c9a640 10 monclient: build_initial_monmap 2024-03-29T11:06:50.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.201+0000 7ffaf3c9a640 10 monclient: monmap: 2024-03-29T11:06:50.182 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:06:50.182 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:50.182 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:06:50.203722+0000 2024-03-29T11:06:50.182 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:06:50.203722+0000 2024-03-29T11:06:50.182 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:50.182 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:50.182 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:06:50.182 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:06:50.182 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:06:50.182 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:50.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.203+0000 7ffaf3c9a640 1 Processor -- start 2024-03-29T11:06:50.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.203+0000 7ffaf3c9a640 1 -- start start 2024-03-29T11:06:50.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.203+0000 7ffaf3c9a640 10 monclient: init 2024-03-29T11:06:50.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.204+0000 7ffaf3c9a640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:50.185 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.204+0000 7ffaf3c9a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:06:50.185 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.204+0000 7ffaf3c9a640 10 monclient(hunting): picked mon.noname-c con 0x7ffaec1422b0 addr v1:172.21.15.137:6789/0 2024-03-29T11:06:50.185 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.204+0000 7ffaf3c9a640 10 monclient(hunting): picked mon.noname-a con 0x7ffaec142690 addr v1:172.21.15.71:6789/0 2024-03-29T11:06:50.185 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.204+0000 7ffaf3c9a640 10 monclient(hunting): picked mon.noname-b con 0x7ffaec0b17d0 addr v1:172.21.15.71:6790/0 2024-03-29T11:06:50.185 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.204+0000 7ffaf3c9a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec142c80 con 0x7ffaec142690 2024-03-29T11:06:50.185 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.204+0000 7ffaf3c9a640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec0aae40 con 0x7ffaec1422b0 2024-03-29T11:06:50.185 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.204+0000 7ffaf3c9a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec13fbd0 con 0x7ffaec0b17d0 2024-03-29T11:06:50.185 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.204+0000 7ffaf3c9a640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:50.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.204+0000 7ffaf3c9a640 10 monclient(hunting): authenticate will time out at 1166.463452s 2024-03-29T11:06:50.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.206+0000 7ffaf1235640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7ffaec142690 0x7ffaec0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:51940/0 (socket says 172.21.15.71:51940) 2024-03-29T11:06:50.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.206+0000 7ffaf1235640 1 -- 172.21.15.71:0/4213753957 learned_addr learned my addr 172.21.15.71:0/4213753957 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:06:50.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.207+0000 7ffaf0a34640 1 -- 172.21.15.71:0/4213753957 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3192286761 0 0) 0x7ffaec142c80 con 0x7ffaec142690 2024-03-29T11:06:50.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.207+0000 7ffaf0a34640 10 monclient(hunting): _negotiate 2024-03-29T11:06:50.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.207+0000 7ffaf0a34640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:50.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.207+0000 7ffaf0a34640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:50.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.207+0000 7ffaf0a34640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:06:50.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.207+0000 7ffaf0a34640 1 -- 172.21.15.71:0/4213753957 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad8003520 con 0x7ffaec142690 2024-03-29T11:06:50.188 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.207+0000 7ffaf0a34640 1 -- 172.21.15.71:0/4213753957 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1939330501 0 0) 0x7ffad8003520 con 0x7ffaec142690 2024-03-29T11:06:50.188 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.207+0000 7ffaf0a34640 1 -- 172.21.15.71:0/4213753957 >> v1:172.21.15.71:6790/0 conn(0x7ffaec0b17d0 legacy=0x7ffaec0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:06:50.188 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.207+0000 7ffaf0a34640 1 -- 172.21.15.71:0/4213753957 >> v1:172.21.15.137:6789/0 conn(0x7ffaec1422b0 legacy=0x7ffaec13f4c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:06:50.188 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.207+0000 7ffaf0a34640 10 monclient: _finish_hunting 0 2024-03-29T11:06:50.188 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.207+0000 7ffaf0a34640 1 monclient: found mon.noname-a 2024-03-29T11:06:50.189 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.207+0000 7ffaf0a34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:50.189 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.207+0000 7ffaf0a34640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:06:50.189 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.207+0000 7ffaf0a34640 1 -- 172.21.15.71:0/4213753957 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaec151940 con 0x7ffaec142690 2024-03-29T11:06:50.189 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.207+0000 7ffaf0a34640 10 monclient: _finish_auth 0 2024-03-29T11:06:50.189 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.207+0000 7ffaf0a34640 10 monclient: _check_auth_tickets 2024-03-29T11:06:50.189 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.207+0000 7ffaf0a34640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:50.189 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.207+0000 7ffaf0a34640 1 -- 172.21.15.71:0/4213753957 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7ffadc001690 con 0x7ffaec142690 2024-03-29T11:06:50.189 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.208+0000 7ffaf0a34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:50.189 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.208+0000 7ffaf0a34640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:06:50.189 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.208+0000 7ffaf0a34640 10 monclient: dump: 2024-03-29T11:06:50.189 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:50.189 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:50.189 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:50.189 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:50.190 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:50.190 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:50.190 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:50.190 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:50.190 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:50.190 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:50.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.208+0000 7ffaf0a34640 1 -- 172.21.15.71:0/4213753957 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffadc0018f0 con 0x7ffaec142690 2024-03-29T11:06:50.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.208+0000 7ffaf0a34640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:50.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.208+0000 7ffaf0a34640 1 -- 172.21.15.71:0/4213753957 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7ffadc001690 con 0x7ffaec142690 2024-03-29T11:06:50.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.208+0000 7ffaf0a34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:50.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.208+0000 7ffaf0a34640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:06:50.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.208+0000 7ffaf0a34640 10 monclient: dump: 2024-03-29T11:06:50.190 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:50.191 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:50.191 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:50.191 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:50.191 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:50.191 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:50.191 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:50.191 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:50.191 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:50.191 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:50.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.208+0000 7ffaf3c9a640 5 monclient: authenticate success, global_id 4461 2024-03-29T11:06:50.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.208+0000 7ffaf3c9a640 10 monclient: get_monmap_and_config success 2024-03-29T11:06:50.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.208+0000 7ffaf3c9a640 4 set_mon_vals no callback set 2024-03-29T11:06:50.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.209+0000 7ffaf3c9a640 10 monclient: shutdown 2024-03-29T11:06:50.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.209+0000 7ffaf3c9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:50.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.209+0000 7ffaf3c9a640 1 -- 172.21.15.71:0/4213753957 >> v1:172.21.15.71:6789/0 conn(0x7ffaec142690 legacy=0x7ffaec0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:50.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.210+0000 7ffaf3c9a640 1 -- 172.21.15.71:0/4213753957 shutdown_connections 2024-03-29T11:06:50.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.210+0000 7ffaf3c9a640 1 -- 172.21.15.71:0/4213753957 >> 172.21.15.71:0/4213753957 conn(0x7ffaec066a00 msgr2=0x7ffaec066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:50.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.210+0000 7ffaf3c9a640 1 -- 172.21.15.71:0/4213753957 shutdown_connections 2024-03-29T11:06:50.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.211+0000 7ffaf3c9a640 1 -- 172.21.15.71:0/4213753957 wait complete. 2024-03-29T11:06:50.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.212+0000 7ffaf3c9a640 10 monclient: build_initial_monmap 2024-03-29T11:06:50.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.212+0000 7ffaf3c9a640 10 monclient: monmap: 2024-03-29T11:06:50.193 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:06:50.193 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:06:50.193 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:06:50.193 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:06:50.194 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:50.194 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:50.194 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:06:50.194 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:06:50.194 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:06:50.194 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:50.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.213+0000 7ffaf3c9a640 1 librados: starting msgr at 2024-03-29T11:06:50.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.213+0000 7ffaf3c9a640 1 librados: starting objecter 2024-03-29T11:06:50.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.213+0000 7ffaf3c9a640 1 Processor -- start 2024-03-29T11:06:50.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.213+0000 7ffaf3c9a640 1 -- start start 2024-03-29T11:06:50.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.213+0000 7ffaf3c9a640 1 librados: setting wanted keys 2024-03-29T11:06:50.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.213+0000 7ffaf3c9a640 1 librados: calling monclient init 2024-03-29T11:06:50.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.213+0000 7ffaf3c9a640 10 monclient: init 2024-03-29T11:06:50.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.213+0000 7ffaf3c9a640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:50.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.213+0000 7ffaf3c9a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:06:50.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.215+0000 7ffaf3c9a640 10 monclient(hunting): picked mon.noname-b con 0x7ffaec0b17d0 addr v1:172.21.15.137:6789/0 2024-03-29T11:06:50.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.215+0000 7ffaf3c9a640 10 monclient(hunting): picked mon.noname-a con 0x7ffaec1422b0 addr v1:172.21.15.71:6789/0 2024-03-29T11:06:50.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.215+0000 7ffaf3c9a640 10 monclient(hunting): picked mon.noname-c con 0x7ffaec142690 addr v1:172.21.15.71:6790/0 2024-03-29T11:06:50.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.215+0000 7ffaf3c9a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec06ebd0 con 0x7ffaec1422b0 2024-03-29T11:06:50.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.215+0000 7ffaf3c9a640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec13cf50 con 0x7ffaec0b17d0 2024-03-29T11:06:50.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.215+0000 7ffaf3c9a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec13e3b0 con 0x7ffaec142690 2024-03-29T11:06:50.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.215+0000 7ffaf3c9a640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:50.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.215+0000 7ffaf3c9a640 10 monclient(hunting): authenticate will time out at 1166.474732s 2024-03-29T11:06:50.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.217+0000 7ffaf1235640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7ffaec1422b0 0x7ffaec071720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:52800/0 (socket says 172.21.15.71:52800) 2024-03-29T11:06:50.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.217+0000 7ffaf1235640 1 -- 172.21.15.71:0/515172811 learned_addr learned my addr 172.21.15.71:0/515172811 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:06:50.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.217+0000 7ffaf1a36640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7ffaec0b17d0 0x7ffaec13dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:60911/0 (socket says 172.21.15.71:60911) 2024-03-29T11:06:50.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.218+0000 7ffae27fc640 10 client.?.objecter ms_handle_connect 0x7ffaec1422b0 2024-03-29T11:06:50.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.218+0000 7ffae27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:50.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.218+0000 7ffae27fc640 10 client.?.objecter ms_handle_connect 0x7ffaec0b17d0 2024-03-29T11:06:50.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.218+0000 7ffae27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:50.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.219+0000 7ffae27fc640 1 -- 172.21.15.71:0/515172811 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345105060 0 0) 0x7ffaec13cf50 con 0x7ffaec0b17d0 2024-03-29T11:06:50.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.219+0000 7ffae27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:06:50.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.219+0000 7ffae27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:50.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.219+0000 7ffae27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:50.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.219+0000 7ffae27fc640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:06:50.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.219+0000 7ffae27fc640 1 -- 172.21.15.71:0/515172811 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad0004eb0 con 0x7ffaec0b17d0 2024-03-29T11:06:50.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.219+0000 7ffae27fc640 1 -- 172.21.15.71:0/515172811 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2159966291 0 0) 0x7ffaec06ebd0 con 0x7ffaec1422b0 2024-03-29T11:06:50.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.219+0000 7ffae27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:06:50.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.219+0000 7ffae27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:50.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.219+0000 7ffae27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:50.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.219+0000 7ffae27fc640 10 monclient(hunting): my global_id is 4464 2024-03-29T11:06:50.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.219+0000 7ffae27fc640 1 -- 172.21.15.71:0/515172811 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaec13cf50 con 0x7ffaec1422b0 2024-03-29T11:06:50.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffae27fc640 1 -- 172.21.15.71:0/515172811 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1508631489 0 0) 0x7ffad0004eb0 con 0x7ffaec0b17d0 2024-03-29T11:06:50.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffae27fc640 1 -- 172.21.15.71:0/515172811 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffaec06ebd0 con 0x7ffaec0b17d0 2024-03-29T11:06:50.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffae27fc640 1 -- 172.21.15.71:0/515172811 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7ffae8001870 con 0x7ffaec0b17d0 2024-03-29T11:06:50.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffae27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:50.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffae27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:06:50.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffae27fc640 10 monclient(hunting): dump: 2024-03-29T11:06:50.202 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:50.202 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:50.202 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:50.203 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:50.203 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:50.203 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:50.203 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:50.203 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:50.203 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:50.203 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:50.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffae27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:06:50.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffae27fc640 1 -- 172.21.15.71:0/515172811 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4073848336 0 0) 0x7ffaec06ebd0 con 0x7ffaec0b17d0 2024-03-29T11:06:50.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffae27fc640 1 -- 172.21.15.71:0/515172811 >> v1:172.21.15.71:6790/0 conn(0x7ffaec142690 legacy=0x7ffaec1e6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:50.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffae27fc640 1 -- 172.21.15.71:0/515172811 >> v1:172.21.15.71:6789/0 conn(0x7ffaec1422b0 legacy=0x7ffaec071720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:50.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffae27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:06:50.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffae27fc640 1 monclient: found mon.b 2024-03-29T11:06:50.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffae27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:50.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffae27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:06:50.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffae27fc640 1 -- 172.21.15.71:0/515172811 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaec071e30 con 0x7ffaec0b17d0 2024-03-29T11:06:50.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffae27fc640 10 monclient: _finish_auth 0 2024-03-29T11:06:50.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffae27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:06:50.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffae27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:50.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffaf3c9a640 5 monclient: authenticate success, global_id 4414 2024-03-29T11:06:50.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffaf3c9a640 10 monclient: _renew_subs 2024-03-29T11:06:50.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffaf3c9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:06:50.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffaf3c9a640 1 -- 172.21.15.71:0/515172811 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaec0720a0 con 0x7ffaec0b17d0 2024-03-29T11:06:50.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffaf3c9a640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:06:50.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffaf3c9a640 10 monclient: _renew_subs 2024-03-29T11:06:50.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffaf3c9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:06:50.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffaf3c9a640 1 -- 172.21.15.71:0/515172811 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaec1dee30 con 0x7ffaec0b17d0 2024-03-29T11:06:50.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffaf3c9a640 1 librados: init done 2024-03-29T11:06:50.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffae27fc640 1 -- 172.21.15.71:0/515172811 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffae8001a50 con 0x7ffaec0b17d0 2024-03-29T11:06:50.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffae27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:50.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.220+0000 7ffae37fe640 4 set_mon_vals no callback set 2024-03-29T11:06:50.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.223+0000 7ffae27fc640 1 -- 172.21.15.71:0/515172811 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7ffae80049e0 con 0x7ffaec0b17d0 2024-03-29T11:06:50.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.223+0000 7ffae27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:50.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.223+0000 7ffae27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:06:50.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.223+0000 7ffae27fc640 10 monclient: dump: 2024-03-29T11:06:50.207 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:50.207 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:50.207 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:50.207 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:50.207 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:50.207 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:50.207 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:50.208 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:50.208 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:50.208 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:50.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.223+0000 7ffae27fc640 1 -- 172.21.15.71:0/515172811 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7ffae8004c00 con 0x7ffaec0b17d0 2024-03-29T11:06:50.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.223+0000 7ffae27fc640 1 -- 172.21.15.71:0/515172811 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7ffae8084a30 con 0x7ffaec0b17d0 2024-03-29T11:06:50.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.223+0000 7ffae27fc640 10 client.4414.objecter ms_dispatch 0x7ffaec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:06:50.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.223+0000 7ffaf3c9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:50.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.223+0000 7ffaf3c9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:50.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.224+0000 7ffaf3c9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:06:50.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.224+0000 7ffaf3c9a640 1 -- 172.21.15.71:0/515172811 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffad4005080 con 0x7ffaec0b17d0 2024-03-29T11:06:50.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.224+0000 7ffaf1235640 10 monclient: get_auth_request con 0x7ffad006db90 auth_method 0 2024-03-29T11:06:50.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.224+0000 7ffae27fc640 3 client.4414.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:06:50.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.224+0000 7ffae27fc640 3 client.4414.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:06:50.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.224+0000 7ffae27fc640 20 client.4414.objecter dump_active .. 0 homeless 2024-03-29T11:06:50.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.224+0000 7ffae27fc640 10 client.4414.objecter ms_handle_connect 0x7ffad006db90 2024-03-29T11:06:50.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.232+0000 7ffae27fc640 1 -- 172.21.15.71:0/515172811 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffae8050390 con 0x7ffaec0b17d0 2024-03-29T11:06:50.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.232+0000 7ffae27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:50.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.232+0000 7ffae27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:06:50.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.553+0000 7ffaf3c9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:50.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.553+0000 7ffaf3c9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:50.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.553+0000 7ffaf3c9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:06:50.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.553+0000 7ffaf3c9a640 1 -- 172.21.15.71:0/515172811 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffad4005330 con 0x7ffaec0b17d0 2024-03-29T11:06:50.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.555+0000 7ffae27fc640 1 -- 172.21.15.71:0/515172811 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7ffae80522c0 con 0x7ffaec0b17d0 2024-03-29T11:06:50.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.555+0000 7ffae27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:50.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.555+0000 7ffae27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:06:50.535 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:06:50.535 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:06:50.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.558+0000 7ffaf3c9a640 10 librados: watch_flush enter 2024-03-29T11:06:50.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.560+0000 7ffaf3c9a640 10 librados: watch_flush exit 2024-03-29T11:06:50.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.560+0000 7ffaf3c9a640 20 client.4414.objecter shutdown clearing up homeless session... 2024-03-29T11:06:50.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.560+0000 7ffaf3c9a640 10 client.4414.objecter successfully canceled tick 2024-03-29T11:06:50.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.560+0000 7ffaf3c9a640 1 -- 172.21.15.71:0/515172811 >> v1:172.21.15.137:6816/973332665 conn(0x7ffad006db90 legacy=0x7ffad0070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:50.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.560+0000 7ffaf3c9a640 10 monclient: shutdown 2024-03-29T11:06:50.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.560+0000 7ffaf3c9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:50.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.560+0000 7ffaf3c9a640 1 -- 172.21.15.71:0/515172811 >> v1:172.21.15.137:6789/0 conn(0x7ffaec0b17d0 legacy=0x7ffaec13dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:50.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.561+0000 7ffaf3c9a640 1 -- 172.21.15.71:0/515172811 shutdown_connections 2024-03-29T11:06:50.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.561+0000 7ffaf3c9a640 1 -- 172.21.15.71:0/515172811 >> 172.21.15.71:0/515172811 conn(0x7ffaec066a00 msgr2=0x7ffaec0b3760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:50.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.561+0000 7ffaf3c9a640 1 -- 172.21.15.71:0/515172811 shutdown_connections 2024-03-29T11:06:50.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.561+0000 7ffaf3c9a640 1 -- 172.21.15.71:0/515172811 wait complete. 2024-03-29T11:06:50.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:50.561+0000 7ffaf3c9a640 1 librados: shutdown 2024-03-29T11:06:50.557 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:06:57.364 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:06:57.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.531+0000 7f8f9ad23640 10 monclient: get_monmap_and_config 2024-03-29T11:06:57.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.531+0000 7f8f9ad23640 10 monclient: build_initial_monmap 2024-03-29T11:06:57.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.531+0000 7f8f9ad23640 10 monclient: monmap: 2024-03-29T11:06:57.510 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:06:57.510 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:57.510 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:06:57.532268+0000 2024-03-29T11:06:57.511 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:06:57.532268+0000 2024-03-29T11:06:57.511 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:57.511 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:57.511 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:06:57.511 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:06:57.511 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:06:57.511 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:57.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.532+0000 7f8f9ad23640 1 Processor -- start 2024-03-29T11:06:57.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.532+0000 7f8f9ad23640 1 -- start start 2024-03-29T11:06:57.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.532+0000 7f8f9ad23640 10 monclient: init 2024-03-29T11:06:57.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.533+0000 7f8f9ad23640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:57.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.533+0000 7f8f9ad23640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:06:57.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.533+0000 7f8f9ad23640 10 monclient(hunting): picked mon.noname-c con 0x7f8f941457f0 addr v1:172.21.15.137:6789/0 2024-03-29T11:06:57.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.533+0000 7f8f9ad23640 10 monclient(hunting): picked mon.noname-b con 0x7f8f9413f650 addr v1:172.21.15.71:6790/0 2024-03-29T11:06:57.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.533+0000 7f8f9ad23640 10 monclient(hunting): picked mon.noname-a con 0x7f8f940a9d70 addr v1:172.21.15.71:6789/0 2024-03-29T11:06:57.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.533+0000 7f8f9ad23640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f940aaef0 con 0x7f8f940a9d70 2024-03-29T11:06:57.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.533+0000 7f8f9ad23640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f94145bd0 con 0x7f8f941457f0 2024-03-29T11:06:57.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.533+0000 7f8f9ad23640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f94145eb0 con 0x7f8f9413f650 2024-03-29T11:06:57.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.533+0000 7f8f9ad23640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:57.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.533+0000 7f8f9ad23640 10 monclient(hunting): authenticate will time out at 1173.791584s 2024-03-29T11:06:57.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.533+0000 7f8f98abf640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f8f941457f0 0x7f8f9413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:54032/0 (socket says 172.21.15.71:54032) 2024-03-29T11:06:57.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.533+0000 7f8f98abf640 1 -- 172.21.15.71:0/125100473 learned_addr learned my addr 172.21.15.71:0/125100473 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:06:57.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.534+0000 7f8f937fe640 1 -- 172.21.15.71:0/125100473 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1098145635 0 0) 0x7f8f94145bd0 con 0x7f8f941457f0 2024-03-29T11:06:57.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.534+0000 7f8f937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:06:57.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.534+0000 7f8f937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:57.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.534+0000 7f8f937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:57.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.534+0000 7f8f937fe640 10 monclient(hunting): my global_id is 4417 2024-03-29T11:06:57.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.534+0000 7f8f937fe640 1 -- 172.21.15.71:0/125100473 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f80003520 con 0x7f8f941457f0 2024-03-29T11:06:57.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.534+0000 7f8f937fe640 1 -- 172.21.15.71:0/125100473 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997145617 0 0) 0x7f8f94145eb0 con 0x7f8f9413f650 2024-03-29T11:06:57.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.534+0000 7f8f937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:06:57.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.534+0000 7f8f937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:57.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.534+0000 7f8f937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:57.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.534+0000 7f8f937fe640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:06:57.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.534+0000 7f8f937fe640 1 -- 172.21.15.71:0/125100473 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f94145bd0 con 0x7f8f9413f650 2024-03-29T11:06:57.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.535+0000 7f8f937fe640 1 -- 172.21.15.71:0/125100473 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526947016 0 0) 0x7f8f940aaef0 con 0x7f8f940a9d70 2024-03-29T11:06:57.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.535+0000 7f8f937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:06:57.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.535+0000 7f8f937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:57.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.535+0000 7f8f937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:57.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.535+0000 7f8f937fe640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:06:57.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.535+0000 7f8f937fe640 1 -- 172.21.15.71:0/125100473 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f94145eb0 con 0x7f8f940a9d70 2024-03-29T11:06:57.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.535+0000 7f8f937fe640 1 -- 172.21.15.71:0/125100473 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1946571002 0 0) 0x7f8f94145bd0 con 0x7f8f9413f650 2024-03-29T11:06:57.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.535+0000 7f8f937fe640 1 -- 172.21.15.71:0/125100473 >> v1:172.21.15.137:6789/0 conn(0x7f8f941457f0 legacy=0x7f8f9413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:57.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.535+0000 7f8f937fe640 1 -- 172.21.15.71:0/125100473 >> v1:172.21.15.71:6789/0 conn(0x7f8f940a9d70 legacy=0x7f8f9414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:57.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.535+0000 7f8f937fe640 10 monclient: _finish_hunting 0 2024-03-29T11:06:57.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.535+0000 7f8f937fe640 1 monclient: found mon.noname-b 2024-03-29T11:06:57.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.535+0000 7f8f937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:57.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.535+0000 7f8f937fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:06:57.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.535+0000 7f8f937fe640 1 -- 172.21.15.71:0/125100473 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f94146340 con 0x7f8f9413f650 2024-03-29T11:06:57.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.535+0000 7f8f937fe640 10 monclient: _finish_auth 0 2024-03-29T11:06:57.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.535+0000 7f8f937fe640 10 monclient: _check_auth_tickets 2024-03-29T11:06:57.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.535+0000 7f8f937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:57.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.535+0000 7f8f9ad23640 5 monclient: authenticate success, global_id 4499 2024-03-29T11:06:57.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.535+0000 7f8f9ad23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:06:57.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.535+0000 7f8f937fe640 1 -- 172.21.15.71:0/125100473 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f8f840019f0 con 0x7f8f9413f650 2024-03-29T11:06:57.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.535+0000 7f8f937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:57.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.535+0000 7f8f937fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:06:57.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.535+0000 7f8f937fe640 10 monclient: dump: 2024-03-29T11:06:57.518 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:57.518 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:57.519 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:57.519 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:57.519 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:57.519 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:57.519 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:57.519 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:57.519 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:57.519 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:57.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.535+0000 7f8f9ad23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:06:57.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.536+0000 7f8f937fe640 1 -- 172.21.15.71:0/125100473 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f84001bd0 con 0x7f8f9413f650 2024-03-29T11:06:57.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.536+0000 7f8f937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:57.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.536+0000 7f8f9ad23640 10 monclient: get_monmap_and_config success 2024-03-29T11:06:57.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.536+0000 7f8f9ad23640 4 set_mon_vals no callback set 2024-03-29T11:06:57.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.536+0000 7f8f937fe640 1 -- 172.21.15.71:0/125100473 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f8f840019f0 con 0x7f8f9413f650 2024-03-29T11:06:57.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.536+0000 7f8f937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:57.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.536+0000 7f8f937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:06:57.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.536+0000 7f8f937fe640 10 monclient: dump: 2024-03-29T11:06:57.520 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:57.520 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:57.520 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:57.520 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:57.520 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:57.520 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:57.520 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:57.520 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:57.520 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:57.520 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:57.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.537+0000 7f8f9ad23640 10 monclient: shutdown 2024-03-29T11:06:57.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.537+0000 7f8f9ad23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:57.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.537+0000 7f8f9ad23640 1 -- 172.21.15.71:0/125100473 >> v1:172.21.15.71:6790/0 conn(0x7f8f9413f650 legacy=0x7f8f940b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:57.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.537+0000 7f8f9ad23640 1 -- 172.21.15.71:0/125100473 shutdown_connections 2024-03-29T11:06:57.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.537+0000 7f8f9ad23640 1 -- 172.21.15.71:0/125100473 >> 172.21.15.71:0/125100473 conn(0x7f8f94066a00 msgr2=0x7f8f94066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:57.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.537+0000 7f8f9ad23640 1 -- 172.21.15.71:0/125100473 shutdown_connections 2024-03-29T11:06:57.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.537+0000 7f8f9ad23640 1 -- 172.21.15.71:0/125100473 wait complete. 2024-03-29T11:06:57.527 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.538+0000 7f8f9ad23640 10 monclient: build_initial_monmap 2024-03-29T11:06:57.527 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.538+0000 7f8f9ad23640 10 monclient: monmap: 2024-03-29T11:06:57.527 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:06:57.527 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:06:57.527 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:06:57.527 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:06:57.527 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:06:57.527 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:57.528 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:06:57.528 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:06:57.528 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:06:57.528 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:57.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.539+0000 7f8f9ad23640 1 librados: starting msgr at 2024-03-29T11:06:57.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.539+0000 7f8f9ad23640 1 librados: starting objecter 2024-03-29T11:06:57.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.539+0000 7f8f9ad23640 1 Processor -- start 2024-03-29T11:06:57.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.539+0000 7f8f9ad23640 1 -- start start 2024-03-29T11:06:57.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.539+0000 7f8f9ad23640 1 librados: setting wanted keys 2024-03-29T11:06:57.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.539+0000 7f8f9ad23640 1 librados: calling monclient init 2024-03-29T11:06:57.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.539+0000 7f8f9ad23640 10 monclient: init 2024-03-29T11:06:57.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.540+0000 7f8f9ad23640 10 monclient: _reopen_session rank -1 2024-03-29T11:06:57.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.540+0000 7f8f9ad23640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:06:57.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.540+0000 7f8f9ad23640 10 monclient(hunting): picked mon.noname-c con 0x7f8f940a9d70 addr v1:172.21.15.71:6790/0 2024-03-29T11:06:57.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.540+0000 7f8f9ad23640 10 monclient(hunting): picked mon.noname-b con 0x7f8f9413f650 addr v1:172.21.15.137:6789/0 2024-03-29T11:06:57.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.540+0000 7f8f9ad23640 10 monclient(hunting): picked mon.noname-a con 0x7f8f941457f0 addr v1:172.21.15.71:6789/0 2024-03-29T11:06:57.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.540+0000 7f8f9ad23640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f941db6c0 con 0x7f8f941457f0 2024-03-29T11:06:57.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.540+0000 7f8f9ad23640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f9406f720 con 0x7f8f9413f650 2024-03-29T11:06:57.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.540+0000 7f8f9ad23640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f9406fa00 con 0x7f8f940a9d70 2024-03-29T11:06:57.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.540+0000 7f8f9ad23640 10 monclient(hunting): _renew_subs 2024-03-29T11:06:57.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.540+0000 7f8f9ad23640 10 monclient(hunting): authenticate will time out at 1173.798786s 2024-03-29T11:06:57.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.540+0000 7f8f992c0640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f8f941457f0 0x7f8f941dbe10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:32986/0 (socket says 172.21.15.71:32986) 2024-03-29T11:06:57.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.540+0000 7f8f992c0640 1 -- 172.21.15.71:0/971329909 learned_addr learned my addr 172.21.15.71:0/971329909 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:06:57.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.541+0000 7f8f917fa640 10 client.?.objecter ms_handle_connect 0x7f8f941457f0 2024-03-29T11:06:57.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.541+0000 7f8f917fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:57.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.541+0000 7f8f917fa640 10 client.?.objecter ms_handle_connect 0x7f8f940a9d70 2024-03-29T11:06:57.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.541+0000 7f8f917fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:57.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.541+0000 7f8f917fa640 1 -- 172.21.15.71:0/971329909 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2004150662 0 0) 0x7f8f941db6c0 con 0x7f8f941457f0 2024-03-29T11:06:57.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.541+0000 7f8f917fa640 10 monclient(hunting): _negotiate 2024-03-29T11:06:57.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.541+0000 7f8f917fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:57.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.541+0000 7f8f917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:57.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.541+0000 7f8f917fa640 10 monclient(hunting): my global_id is 4470 2024-03-29T11:06:57.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.541+0000 7f8f917fa640 1 -- 172.21.15.71:0/971329909 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f74004eb0 con 0x7f8f941457f0 2024-03-29T11:06:57.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.541+0000 7f8f917fa640 1 -- 172.21.15.71:0/971329909 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2126286832 0 0) 0x7f8f74004eb0 con 0x7f8f941457f0 2024-03-29T11:06:57.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.541+0000 7f8f917fa640 1 -- 172.21.15.71:0/971329909 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f941db6c0 con 0x7f8f941457f0 2024-03-29T11:06:57.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.541+0000 7f8f917fa640 1 -- 172.21.15.71:0/971329909 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f8f8c001990 con 0x7f8f941457f0 2024-03-29T11:06:57.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.541+0000 7f8f917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:57.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.541+0000 7f8f917fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:06:57.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.541+0000 7f8f917fa640 10 monclient(hunting): dump: 2024-03-29T11:06:57.530 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:57.530 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:57.530 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:57.530 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:57.531 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:57.531 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:57.531 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:57.531 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:57.531 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:57.531 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:57.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.541+0000 7f8f917fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:06:57.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.541+0000 7f8f917fa640 1 -- 172.21.15.71:0/971329909 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1357820178 0 0) 0x7f8f9406fa00 con 0x7f8f940a9d70 2024-03-29T11:06:57.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.541+0000 7f8f917fa640 10 monclient(hunting): _negotiate 2024-03-29T11:06:57.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.541+0000 7f8f917fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:06:57.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:06:57.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f917fa640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:06:57.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f917fa640 1 -- 172.21.15.71:0/971329909 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f74004eb0 con 0x7f8f940a9d70 2024-03-29T11:06:57.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f917fa640 10 client.?.objecter ms_handle_connect 0x7f8f9413f650 2024-03-29T11:06:57.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f917fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:06:57.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f917fa640 1 -- 172.21.15.71:0/971329909 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3280051640 0 0) 0x7f8f941db6c0 con 0x7f8f941457f0 2024-03-29T11:06:57.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f917fa640 1 -- 172.21.15.71:0/971329909 >> v1:172.21.15.71:6790/0 conn(0x7f8f940a9d70 legacy=0x7f8f9413b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:57.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f917fa640 1 -- 172.21.15.71:0/971329909 >> v1:172.21.15.137:6789/0 conn(0x7f8f9413f650 legacy=0x7f8f94071860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:57.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f917fa640 10 monclient: _finish_hunting 0 2024-03-29T11:06:57.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f917fa640 1 monclient: found mon.a 2024-03-29T11:06:57.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f917fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:06:57.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f917fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:06:57.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f917fa640 1 -- 172.21.15.71:0/971329909 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f94071f70 con 0x7f8f941457f0 2024-03-29T11:06:57.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f917fa640 10 monclient: _finish_auth 0 2024-03-29T11:06:57.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f917fa640 10 monclient: _check_auth_tickets 2024-03-29T11:06:57.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:06:57.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f9ad23640 5 monclient: authenticate success, global_id 4470 2024-03-29T11:06:57.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f9ad23640 10 monclient: _renew_subs 2024-03-29T11:06:57.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f9ad23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:06:57.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f9ad23640 1 -- 172.21.15.71:0/971329909 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f941dc520 con 0x7f8f941457f0 2024-03-29T11:06:57.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f9ad23640 10 client.4470.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:06:57.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f9ad23640 10 monclient: _renew_subs 2024-03-29T11:06:57.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f9ad23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:06:57.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f9ad23640 1 -- 172.21.15.71:0/971329909 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f94138cc0 con 0x7f8f941457f0 2024-03-29T11:06:57.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f9ad23640 1 librados: init done 2024-03-29T11:06:57.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f917fa640 1 -- 172.21.15.71:0/971329909 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f8c001af0 con 0x7f8f941457f0 2024-03-29T11:06:57.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f917fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:06:57.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f917fa640 1 -- 172.21.15.71:0/971329909 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f8f8c003a60 con 0x7f8f941457f0 2024-03-29T11:06:57.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:06:57.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f92ffd640 4 set_mon_vals no callback set 2024-03-29T11:06:57.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.542+0000 7f8f917fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:06:57.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.543+0000 7f8f917fa640 10 monclient: dump: 2024-03-29T11:06:57.534 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:06:57.534 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:06:57.534 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:57.534 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:06:57.534 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:06:57.534 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:06:57.534 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:06:57.534 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:06:57.534 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:06:57.534 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:06:57.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.544+0000 7f8f9ad23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:57.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.544+0000 7f8f9ad23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:57.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.544+0000 7f8f9ad23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:06:57.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.544+0000 7f8f9ad23640 1 -- 172.21.15.71:0/971329909 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f78005080 con 0x7f8f941457f0 2024-03-29T11:06:57.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.545+0000 7f8f917fa640 1 -- 172.21.15.71:0/971329909 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f8f8c003bc0 con 0x7f8f941457f0 2024-03-29T11:06:57.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.546+0000 7f8f917fa640 1 -- 172.21.15.71:0/971329909 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f8f8c084a30 con 0x7f8f941457f0 2024-03-29T11:06:57.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.546+0000 7f8f917fa640 10 client.4470.objecter ms_dispatch 0x7f8f94000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:06:57.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.546+0000 7f8f917fa640 3 client.4470.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:06:57.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.546+0000 7f8f917fa640 3 client.4470.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:06:57.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.546+0000 7f8f98abf640 10 monclient: get_auth_request con 0x7f8f7406ddc0 auth_method 0 2024-03-29T11:06:57.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.546+0000 7f8f917fa640 20 client.4470.objecter dump_active .. 0 homeless 2024-03-29T11:06:57.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.547+0000 7f8f917fa640 10 client.4470.objecter ms_handle_connect 0x7f8f7406ddc0 2024-03-29T11:06:57.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.559+0000 7f8f917fa640 1 -- 172.21.15.71:0/971329909 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8f8c050310 con 0x7f8f941457f0 2024-03-29T11:06:57.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.559+0000 7f8f917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:06:57.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.559+0000 7f8f917fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:06:57.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.921+0000 7f8f9ad23640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:57.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.921+0000 7f8f9ad23640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:57.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.921+0000 7f8f9ad23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:06:57.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.921+0000 7f8f9ad23640 1 -- 172.21.15.71:0/971329909 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f78005330 con 0x7f8f941457f0 2024-03-29T11:06:57.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.923+0000 7f8f917fa640 1 -- 172.21.15.71:0/971329909 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f8f8c052240 con 0x7f8f941457f0 2024-03-29T11:06:57.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.923+0000 7f8f917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:06:57.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.923+0000 7f8f917fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:06:57.902 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:06:57.902 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:06:57.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.926+0000 7f8f9ad23640 10 librados: watch_flush enter 2024-03-29T11:06:57.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.926+0000 7f8f9ad23640 10 librados: watch_flush exit 2024-03-29T11:06:57.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.926+0000 7f8f9ad23640 20 client.4470.objecter shutdown clearing up homeless session... 2024-03-29T11:06:57.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.926+0000 7f8f9ad23640 10 client.4470.objecter successfully canceled tick 2024-03-29T11:06:57.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.926+0000 7f8f9ad23640 1 -- 172.21.15.71:0/971329909 >> v1:172.21.15.137:6816/973332665 conn(0x7f8f7406ddc0 legacy=0x7f8f74070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:57.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.926+0000 7f8f9ad23640 10 monclient: shutdown 2024-03-29T11:06:57.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.926+0000 7f8f9ad23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:06:57.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.926+0000 7f8f9ad23640 1 -- 172.21.15.71:0/971329909 >> v1:172.21.15.71:6789/0 conn(0x7f8f941457f0 legacy=0x7f8f941dbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:06:57.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.926+0000 7f8f9ad23640 1 -- 172.21.15.71:0/971329909 shutdown_connections 2024-03-29T11:06:57.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.926+0000 7f8f9ad23640 1 -- 172.21.15.71:0/971329909 >> 172.21.15.71:0/971329909 conn(0x7f8f94066a00 msgr2=0x7f8f94066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:06:57.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.927+0000 7f8f9ad23640 1 -- 172.21.15.71:0/971329909 shutdown_connections 2024-03-29T11:06:57.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.927+0000 7f8f9ad23640 1 -- 172.21.15.71:0/971329909 wait complete. 2024-03-29T11:06:57.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:06:57.927+0000 7f8f9ad23640 1 librados: shutdown 2024-03-29T11:06:57.920 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:07:04.727 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:07:04.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.903+0000 7f0c15dc3640 10 monclient: get_monmap_and_config 2024-03-29T11:07:04.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.903+0000 7f0c15dc3640 10 monclient: build_initial_monmap 2024-03-29T11:07:04.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.903+0000 7f0c15dc3640 10 monclient: monmap: 2024-03-29T11:07:04.883 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:07:04.883 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:04.883 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:07:04.904661+0000 2024-03-29T11:07:04.883 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:07:04.904661+0000 2024-03-29T11:07:04.883 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:04.883 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:04.883 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:07:04.883 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:07:04.883 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:07:04.883 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:04.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.905+0000 7f0c15dc3640 1 Processor -- start 2024-03-29T11:07:04.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.905+0000 7f0c15dc3640 1 -- start start 2024-03-29T11:07:04.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.905+0000 7f0c15dc3640 10 monclient: init 2024-03-29T11:07:04.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.906+0000 7f0c15dc3640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:04.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.906+0000 7f0c15dc3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:07:04.890 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.909+0000 7f0c15dc3640 10 monclient(hunting): picked mon.noname-c con 0x7f0c08006e30 addr v1:172.21.15.137:6789/0 2024-03-29T11:07:04.890 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.909+0000 7f0c15dc3640 10 monclient(hunting): picked mon.noname-b con 0x7f0c08035e00 addr v1:172.21.15.71:6790/0 2024-03-29T11:07:04.890 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.909+0000 7f0c15dc3640 10 monclient(hunting): picked mon.noname-a con 0x7f0c080c9670 addr v1:172.21.15.71:6789/0 2024-03-29T11:07:04.890 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.909+0000 7f0c15dc3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c08035970 con 0x7f0c080c9670 2024-03-29T11:07:04.890 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.909+0000 7f0c15dc3640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c08036280 con 0x7f0c08006e30 2024-03-29T11:07:04.890 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.909+0000 7f0c15dc3640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c08035640 con 0x7f0c08035e00 2024-03-29T11:07:04.890 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.909+0000 7f0c15dc3640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:04.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.909+0000 7f0c15dc3640 10 monclient(hunting): authenticate will time out at 1181.167230s 2024-03-29T11:07:04.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.910+0000 7f0c14dc1640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f0c08006e30 0x7f0c080cbcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:54311/0 (socket says 172.21.15.71:54311) 2024-03-29T11:07:04.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.910+0000 7f0c14dc1640 1 -- 172.21.15.71:0/162920014 learned_addr learned my addr 172.21.15.71:0/162920014 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:07:04.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.911+0000 7f0c0f7fe640 1 -- 172.21.15.71:0/162920014 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293855551 0 0) 0x7f0c08036280 con 0x7f0c08006e30 2024-03-29T11:07:04.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.911+0000 7f0c0f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:07:04.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.911+0000 7f0c0f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:04.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.911+0000 7f0c0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:04.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.911+0000 7f0c0f7fe640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:07:04.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.911+0000 7f0c0f7fe640 1 -- 172.21.15.71:0/162920014 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c00003520 con 0x7f0c08006e30 2024-03-29T11:07:04.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.911+0000 7f0c0f7fe640 1 -- 172.21.15.71:0/162920014 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342397701 0 0) 0x7f0c08035640 con 0x7f0c08035e00 2024-03-29T11:07:04.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.911+0000 7f0c0f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:07:04.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.911+0000 7f0c0f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:04.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.911+0000 7f0c0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:04.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.911+0000 7f0c0f7fe640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:07:04.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.911+0000 7f0c0f7fe640 1 -- 172.21.15.71:0/162920014 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c08036280 con 0x7f0c08035e00 2024-03-29T11:07:04.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.911+0000 7f0c0f7fe640 1 -- 172.21.15.71:0/162920014 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2742258448 0 0) 0x7f0c00003520 con 0x7f0c08006e30 2024-03-29T11:07:04.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.911+0000 7f0c0f7fe640 1 -- 172.21.15.71:0/162920014 >> v1:172.21.15.71:6790/0 conn(0x7f0c08035e00 legacy=0x7f0c080c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:04.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.911+0000 7f0c0f7fe640 1 -- 172.21.15.71:0/162920014 >> v1:172.21.15.71:6789/0 conn(0x7f0c080c9670 legacy=0x7f0c080d7560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:07:04.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.913+0000 7f0c0f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:07:04.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.913+0000 7f0c0f7fe640 1 monclient: found mon.noname-c 2024-03-29T11:07:04.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.913+0000 7f0c0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:04.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.913+0000 7f0c0f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.137:6789/0 2024-03-29T11:07:04.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.913+0000 7f0c0f7fe640 1 -- 172.21.15.71:0/162920014 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c08034e20 con 0x7f0c08006e30 2024-03-29T11:07:04.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.913+0000 7f0c0f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:07:04.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.913+0000 7f0c0f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:07:04.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.913+0000 7f0c0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:04.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.913+0000 7f0c0f7fe640 1 -- 172.21.15.71:0/162920014 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f0c04001690 con 0x7f0c08006e30 2024-03-29T11:07:04.894 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.913+0000 7f0c0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:04.894 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.914+0000 7f0c0f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:07:04.894 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.914+0000 7f0c0f7fe640 10 monclient: dump: 2024-03-29T11:07:04.894 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:04.894 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:04.894 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:04.894 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:04.894 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:04.894 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:04.894 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:04.894 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:04.894 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:04.894 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:04.894 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.914+0000 7f0c15dc3640 5 monclient: authenticate success, global_id 4423 2024-03-29T11:07:04.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.914+0000 7f0c15dc3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:07:04.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.914+0000 7f0c0f7fe640 1 -- 172.21.15.71:0/162920014 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c04001860 con 0x7f0c08006e30 2024-03-29T11:07:04.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.914+0000 7f0c0f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:04.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.914+0000 7f0c0f7fe640 1 -- 172.21.15.71:0/162920014 <== mon.1 v1:172.21.15.137:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f0c04001cb0 con 0x7f0c08006e30 2024-03-29T11:07:04.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.914+0000 7f0c0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:04.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.914+0000 7f0c0f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:07:04.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.914+0000 7f0c0f7fe640 10 monclient: dump: 2024-03-29T11:07:04.895 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:04.896 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:04.896 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:04.896 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:04.896 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:04.896 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:04.896 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:04.896 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:04.896 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:04.896 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:04.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.914+0000 7f0c15dc3640 10 monclient: get_monmap_and_config success 2024-03-29T11:07:04.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.914+0000 7f0c15dc3640 4 set_mon_vals no callback set 2024-03-29T11:07:04.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.915+0000 7f0c15dc3640 10 monclient: shutdown 2024-03-29T11:07:04.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.915+0000 7f0c15dc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:04.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.915+0000 7f0c15dc3640 1 -- 172.21.15.71:0/162920014 >> v1:172.21.15.137:6789/0 conn(0x7f0c08006e30 legacy=0x7f0c080cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:04.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.915+0000 7f0c15dc3640 1 -- 172.21.15.71:0/162920014 shutdown_connections 2024-03-29T11:07:04.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.915+0000 7f0c15dc3640 1 -- 172.21.15.71:0/162920014 >> 172.21.15.71:0/162920014 conn(0x7f0c08005c00 msgr2=0x7f0c08006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:04.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.918+0000 7f0c15dc3640 1 -- 172.21.15.71:0/162920014 shutdown_connections 2024-03-29T11:07:04.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.919+0000 7f0c15dc3640 1 -- 172.21.15.71:0/162920014 wait complete. 2024-03-29T11:07:04.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.920+0000 7f0c15dc3640 10 monclient: build_initial_monmap 2024-03-29T11:07:04.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.920+0000 7f0c15dc3640 10 monclient: monmap: 2024-03-29T11:07:04.900 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:07:04.900 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:07:04.900 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:07:04.900 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:07:04.900 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:04.900 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:04.900 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:07:04.900 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:07:04.900 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:07:04.900 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:04.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.920+0000 7f0c15dc3640 1 librados: starting msgr at 2024-03-29T11:07:04.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.920+0000 7f0c15dc3640 1 librados: starting objecter 2024-03-29T11:07:04.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.920+0000 7f0c15dc3640 1 Processor -- start 2024-03-29T11:07:04.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.921+0000 7f0c15dc3640 1 -- start start 2024-03-29T11:07:04.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.921+0000 7f0c15dc3640 1 librados: setting wanted keys 2024-03-29T11:07:04.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.921+0000 7f0c15dc3640 1 librados: calling monclient init 2024-03-29T11:07:04.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.921+0000 7f0c15dc3640 10 monclient: init 2024-03-29T11:07:04.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.922+0000 7f0c15dc3640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:04.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.922+0000 7f0c15dc3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:07:04.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.922+0000 7f0c15dc3640 10 monclient(hunting): picked mon.noname-c con 0x7f0c08006e30 addr v1:172.21.15.71:6790/0 2024-03-29T11:07:04.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.925+0000 7f0c14dc1640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f0c08006e30 0x7f0c080cd2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:42832/0 (socket says 172.21.15.71:42832) 2024-03-29T11:07:04.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.925+0000 7f0c14dc1640 1 -- 172.21.15.71:0/1876179338 learned_addr learned my addr 172.21.15.71:0/1876179338 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:07:04.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.925+0000 7f0c15dc3640 10 monclient(hunting): picked mon.noname-b con 0x7f0c08035e00 addr v1:172.21.15.137:6789/0 2024-03-29T11:07:04.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.925+0000 7f0c0d7fa640 10 client.?.objecter ms_handle_connect 0x7f0c08006e30 2024-03-29T11:07:04.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.925+0000 7f0c0d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:04.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.928+0000 7f0c15dc3640 10 monclient(hunting): picked mon.noname-a con 0x7f0c080c9670 addr v1:172.21.15.71:6789/0 2024-03-29T11:07:04.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.928+0000 7f0c15dc3640 1 -- 172.21.15.71:0/1876179338 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c080ce260 con 0x7f0c080c9670 2024-03-29T11:07:04.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.928+0000 7f0c15dc3640 1 -- 172.21.15.71:0/1876179338 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c080d00c0 con 0x7f0c08035e00 2024-03-29T11:07:04.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.928+0000 7f0c15dc3640 1 -- 172.21.15.71:0/1876179338 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c080ce490 con 0x7f0c08006e30 2024-03-29T11:07:04.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.928+0000 7f0c15dc3640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:04.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.928+0000 7f0c15dc3640 10 monclient(hunting): authenticate will time out at 1181.186902s 2024-03-29T11:07:04.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.929+0000 7f0c0d7fa640 10 client.?.objecter ms_handle_connect 0x7f0c08035e00 2024-03-29T11:07:04.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.929+0000 7f0c0d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:04.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.930+0000 7f0c0d7fa640 1 -- 172.21.15.71:0/1876179338 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303812516 0 0) 0x7f0c080d00c0 con 0x7f0c08035e00 2024-03-29T11:07:04.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.930+0000 7f0c0d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:07:04.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.930+0000 7f0c0d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:04.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.930+0000 7f0c0d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:04.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.930+0000 7f0c0d7fa640 10 monclient(hunting): my global_id is 4426 2024-03-29T11:07:04.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.930+0000 7f0c0d7fa640 1 -- 172.21.15.71:0/1876179338 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bf4004eb0 con 0x7f0c08035e00 2024-03-29T11:07:04.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.931+0000 7f0c0d7fa640 1 -- 172.21.15.71:0/1876179338 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1210416282 0 0) 0x7f0bf4004eb0 con 0x7f0c08035e00 2024-03-29T11:07:04.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.932+0000 7f0c0d7fa640 1 -- 172.21.15.71:0/1876179338 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c080d00c0 con 0x7f0c08035e00 2024-03-29T11:07:04.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.933+0000 7f0c0d7fa640 1 -- 172.21.15.71:0/1876179338 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f0c1004fdd0 con 0x7f0c08035e00 2024-03-29T11:07:04.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.933+0000 7f0c0d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:04.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.934+0000 7f0c0d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:07:04.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.934+0000 7f0c0d7fa640 10 monclient(hunting): dump: 2024-03-29T11:07:04.913 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:04.913 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:04.913 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:04.913 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:04.913 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:04.913 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:04.913 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:04.913 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:04.913 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:04.913 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:04.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.934+0000 7f0c0d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:07:04.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.934+0000 7f0c0d7fa640 1 -- 172.21.15.71:0/1876179338 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2718230788 0 0) 0x7f0c080ce490 con 0x7f0c08006e30 2024-03-29T11:07:04.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.934+0000 7f0c0d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:07:04.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.934+0000 7f0c0d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:04.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.934+0000 7f0c0d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:04.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.934+0000 7f0c0d7fa640 10 monclient(hunting): my global_id is 4508 2024-03-29T11:07:04.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.934+0000 7f0c0d7fa640 1 -- 172.21.15.71:0/1876179338 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bf4004eb0 con 0x7f0c08006e30 2024-03-29T11:07:04.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.934+0000 7f0c0d7fa640 10 client.?.objecter ms_handle_connect 0x7f0c080c9670 2024-03-29T11:07:04.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.934+0000 7f0c0d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:04.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.934+0000 7f0c0d7fa640 1 -- 172.21.15.71:0/1876179338 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2636449624 0 0) 0x7f0bf4004eb0 con 0x7f0c08006e30 2024-03-29T11:07:04.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.934+0000 7f0c0d7fa640 1 -- 172.21.15.71:0/1876179338 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c080ce490 con 0x7f0c08006e30 2024-03-29T11:07:04.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.935+0000 7f0c0d7fa640 1 -- 172.21.15.71:0/1876179338 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f0c040019d0 con 0x7f0c08006e30 2024-03-29T11:07:04.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.935+0000 7f0c0d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:04.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.935+0000 7f0c0d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:07:04.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.935+0000 7f0c0d7fa640 10 monclient(hunting): dump: 2024-03-29T11:07:04.915 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:04.915 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:04.916 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:04.916 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:04.916 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:04.916 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:04.916 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:04.916 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:04.916 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:04.916 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:04.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.935+0000 7f0c0d7fa640 1 -- 172.21.15.71:0/1876179338 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2947401979 0 0) 0x7f0c080d00c0 con 0x7f0c08035e00 2024-03-29T11:07:04.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.935+0000 7f0c0d7fa640 1 -- 172.21.15.71:0/1876179338 >> v1:172.21.15.71:6790/0 conn(0x7f0c08006e30 legacy=0x7f0c080cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:04.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.935+0000 7f0c0d7fa640 1 -- 172.21.15.71:0/1876179338 >> v1:172.21.15.71:6789/0 conn(0x7f0c080c9670 legacy=0x7f0c0816b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:04.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.935+0000 7f0c0d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:07:04.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.935+0000 7f0c0d7fa640 1 monclient: found mon.b 2024-03-29T11:07:04.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.935+0000 7f0c0d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:04.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.935+0000 7f0c0d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:07:04.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.935+0000 7f0c0d7fa640 1 -- 172.21.15.71:0/1876179338 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c080ce770 con 0x7f0c08035e00 2024-03-29T11:07:04.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.935+0000 7f0c0d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:07:04.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.935+0000 7f0c0d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:07:04.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.935+0000 7f0c0d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:04.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.935+0000 7f0c15dc3640 5 monclient: authenticate success, global_id 4426 2024-03-29T11:07:04.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.935+0000 7f0c15dc3640 10 monclient: _renew_subs 2024-03-29T11:07:04.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.935+0000 7f0c15dc3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:07:04.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.935+0000 7f0c15dc3640 1 -- 172.21.15.71:0/1876179338 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c080cea50 con 0x7f0c08035e00 2024-03-29T11:07:04.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.935+0000 7f0c15dc3640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:07:04.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.935+0000 7f0c15dc3640 10 monclient: _renew_subs 2024-03-29T11:07:04.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.935+0000 7f0c15dc3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:07:04.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.935+0000 7f0c15dc3640 1 -- 172.21.15.71:0/1876179338 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c0802f9d0 con 0x7f0c08035e00 2024-03-29T11:07:04.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.935+0000 7f0c15dc3640 1 librados: init done 2024-03-29T11:07:04.918 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.937+0000 7f0c0d7fa640 1 -- 172.21.15.71:0/1876179338 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c1004f400 con 0x7f0c08035e00 2024-03-29T11:07:04.918 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.937+0000 7f0c0d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:04.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.937+0000 7f0c0d7fa640 1 -- 172.21.15.71:0/1876179338 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f0c10075f10 con 0x7f0c08035e00 2024-03-29T11:07:04.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.937+0000 7f0c0d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:04.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.937+0000 7f0c0d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:07:04.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.938+0000 7f0c0e7fc640 4 set_mon_vals no callback set 2024-03-29T11:07:04.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.938+0000 7f0c15dc3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:04.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.937+0000 7f0c0d7fa640 10 monclient: dump: 2024-03-29T11:07:04.919 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:04.919 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:04.919 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:04.919 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:04.919 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:04.919 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:04.919 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:04.919 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:04.920 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:04.920 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:04.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.939+0000 7f0c0d7fa640 1 -- 172.21.15.71:0/1876179338 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f0c10067bf0 con 0x7f0c08035e00 2024-03-29T11:07:04.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.939+0000 7f0c15dc3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:04.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.939+0000 7f0c15dc3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:07:04.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.939+0000 7f0c15dc3640 1 -- 172.21.15.71:0/1876179338 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bf0005080 con 0x7f0c08035e00 2024-03-29T11:07:04.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.939+0000 7f0c0d7fa640 1 -- 172.21.15.71:0/1876179338 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f0c100fe240 con 0x7f0c08035e00 2024-03-29T11:07:04.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.939+0000 7f0c0d7fa640 10 client.4426.objecter ms_dispatch 0x7f0c10000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:07:04.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.939+0000 7f0c0d7fa640 3 client.4426.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:07:04.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.939+0000 7f0c0d7fa640 3 client.4426.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:07:04.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.939+0000 7f0c0d7fa640 20 client.4426.objecter dump_active .. 0 homeless 2024-03-29T11:07:04.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.939+0000 7f0c14dc1640 10 monclient: get_auth_request con 0x7f0bf406dc00 auth_method 0 2024-03-29T11:07:04.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.940+0000 7f0c0d7fa640 10 client.4426.objecter ms_handle_connect 0x7f0bf406dc00 2024-03-29T11:07:04.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.946+0000 7f0c0d7fa640 1 -- 172.21.15.71:0/1876179338 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0c100c9b20 con 0x7f0c08035e00 2024-03-29T11:07:04.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.946+0000 7f0c0d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:04.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:04.946+0000 7f0c0d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:07:05.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:05.314+0000 7f0c15dc3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:05.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:05.314+0000 7f0c15dc3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:05.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:05.314+0000 7f0c15dc3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:07:05.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:05.314+0000 7f0c15dc3640 1 -- 172.21.15.71:0/1876179338 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0bf0005330 con 0x7f0c08035e00 2024-03-29T11:07:05.294 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:05.315+0000 7f0c0d7fa640 1 -- 172.21.15.71:0/1876179338 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f0c100cba50 con 0x7f0c08035e00 2024-03-29T11:07:05.294 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:05.315+0000 7f0c0d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:05.294 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:05.315+0000 7f0c0d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:07:05.294 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:07:05.294 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:07:05.298 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:05.319+0000 7f0c15dc3640 10 librados: watch_flush enter 2024-03-29T11:07:05.298 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:05.319+0000 7f0c15dc3640 10 librados: watch_flush exit 2024-03-29T11:07:05.298 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:05.319+0000 7f0c15dc3640 20 client.4426.objecter shutdown clearing up homeless session... 2024-03-29T11:07:05.298 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:05.319+0000 7f0c15dc3640 10 client.4426.objecter successfully canceled tick 2024-03-29T11:07:05.300 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:05.321+0000 7f0c15dc3640 1 -- 172.21.15.71:0/1876179338 >> v1:172.21.15.137:6816/973332665 conn(0x7f0bf406dc00 legacy=0x7f0bf40700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:05.300 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:05.321+0000 7f0c15dc3640 10 monclient: shutdown 2024-03-29T11:07:05.300 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:05.321+0000 7f0c15dc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:05.300 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:05.321+0000 7f0c15dc3640 1 -- 172.21.15.71:0/1876179338 >> v1:172.21.15.137:6789/0 conn(0x7f0c08035e00 legacy=0x7f0c080cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:05.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:05.322+0000 7f0c15dc3640 1 -- 172.21.15.71:0/1876179338 shutdown_connections 2024-03-29T11:07:05.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:05.322+0000 7f0c15dc3640 1 -- 172.21.15.71:0/1876179338 >> 172.21.15.71:0/1876179338 conn(0x7f0c08005c00 msgr2=0x7f0c080c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:05.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:05.324+0000 7f0c15dc3640 1 -- 172.21.15.71:0/1876179338 shutdown_connections 2024-03-29T11:07:05.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:05.324+0000 7f0c15dc3640 1 -- 172.21.15.71:0/1876179338 wait complete. 2024-03-29T11:07:05.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:05.325+0000 7f0c15dc3640 1 librados: shutdown 2024-03-29T11:07:05.322 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:07:12.130 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:07:13.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.100+0000 7f7e20df3640 10 monclient: get_monmap_and_config 2024-03-29T11:07:13.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.100+0000 7f7e20df3640 10 monclient: build_initial_monmap 2024-03-29T11:07:13.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.100+0000 7f7e20df3640 10 monclient: monmap: 2024-03-29T11:07:13.079 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:07:13.079 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:13.079 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:07:13.100976+0000 2024-03-29T11:07:13.080 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:07:13.100976+0000 2024-03-29T11:07:13.080 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:13.080 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:13.080 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:07:13.080 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:07:13.080 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:07:13.080 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:13.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.101+0000 7f7e20df3640 1 Processor -- start 2024-03-29T11:07:13.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.105+0000 7f7e20df3640 1 -- start start 2024-03-29T11:07:13.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.105+0000 7f7e20df3640 10 monclient: init 2024-03-29T11:07:13.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.106+0000 7f7e20df3640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:13.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.106+0000 7f7e20df3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:07:13.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.106+0000 7f7e20df3640 10 monclient(hunting): picked mon.noname-c con 0x7f7e1c0b2800 addr v1:172.21.15.137:6789/0 2024-03-29T11:07:13.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.106+0000 7f7e20df3640 10 monclient(hunting): picked mon.noname-a con 0x7f7e1c0a9c00 addr v1:172.21.15.71:6789/0 2024-03-29T11:07:13.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.106+0000 7f7e20df3640 10 monclient(hunting): picked mon.noname-b con 0x7f7e1c0b38c0 addr v1:172.21.15.71:6790/0 2024-03-29T11:07:13.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.106+0000 7f7e20df3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e1c141eb0 con 0x7f7e1c0a9c00 2024-03-29T11:07:13.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.106+0000 7f7e20df3640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e1c0b22d0 con 0x7f7e1c0b2800 2024-03-29T11:07:13.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.106+0000 7f7e20df3640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e1c142d30 con 0x7f7e1c0b38c0 2024-03-29T11:07:13.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.106+0000 7f7e20df3640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:13.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.106+0000 7f7e20df3640 10 monclient(hunting): authenticate will time out at 1189.364264s 2024-03-29T11:07:13.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.107+0000 7f7e1bfff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f7e1c0b38c0 0x7f7e1c14d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:40420/0 (socket says 172.21.15.71:40420) 2024-03-29T11:07:13.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.107+0000 7f7e1bfff640 1 -- 172.21.15.71:0/2120384431 learned_addr learned my addr 172.21.15.71:0/2120384431 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:07:13.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.110+0000 7f7e1affd640 1 -- 172.21.15.71:0/2120384431 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3306166340 0 0) 0x7f7e1c142d30 con 0x7f7e1c0b38c0 2024-03-29T11:07:13.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.110+0000 7f7e1affd640 10 monclient(hunting): _negotiate 2024-03-29T11:07:13.100 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.110+0000 7f7e1affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:13.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.110+0000 7f7e1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:13.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.110+0000 7f7e1affd640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:07:13.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.110+0000 7f7e1affd640 1 -- 172.21.15.71:0/2120384431 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e08003520 con 0x7f7e1c0b38c0 2024-03-29T11:07:13.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.110+0000 7f7e1affd640 1 -- 172.21.15.71:0/2120384431 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 203755941 0 0) 0x7f7e1c0b22d0 con 0x7f7e1c0b2800 2024-03-29T11:07:13.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.110+0000 7f7e1affd640 10 monclient(hunting): _negotiate 2024-03-29T11:07:13.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.110+0000 7f7e1affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:13.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.110+0000 7f7e1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:13.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.110+0000 7f7e1affd640 10 monclient(hunting): my global_id is 4429 2024-03-29T11:07:13.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.110+0000 7f7e1affd640 1 -- 172.21.15.71:0/2120384431 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e1c142d30 con 0x7f7e1c0b2800 2024-03-29T11:07:13.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.110+0000 7f7e1affd640 1 -- 172.21.15.71:0/2120384431 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3493467794 0 0) 0x7f7e08003520 con 0x7f7e1c0b38c0 2024-03-29T11:07:13.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.110+0000 7f7e1affd640 1 -- 172.21.15.71:0/2120384431 >> v1:172.21.15.137:6789/0 conn(0x7f7e1c0b2800 legacy=0x7f7e1c142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:13.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.110+0000 7f7e1affd640 1 -- 172.21.15.71:0/2120384431 >> v1:172.21.15.71:6789/0 conn(0x7f7e1c0a9c00 legacy=0x7f7e1c13bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:13.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.110+0000 7f7e1affd640 10 monclient: _finish_hunting 0 2024-03-29T11:07:13.101 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.110+0000 7f7e1affd640 1 monclient: found mon.noname-b 2024-03-29T11:07:13.102 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.110+0000 7f7e1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:13.102 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.110+0000 7f7e1affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:07:13.102 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.110+0000 7f7e1affd640 1 -- 172.21.15.71:0/2120384431 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1c0aaf90 con 0x7f7e1c0b38c0 2024-03-29T11:07:13.102 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.111+0000 7f7e1affd640 10 monclient: _finish_auth 0 2024-03-29T11:07:13.102 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.111+0000 7f7e1affd640 10 monclient: _check_auth_tickets 2024-03-29T11:07:13.102 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.111+0000 7f7e1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:13.102 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.111+0000 7f7e20df3640 5 monclient: authenticate success, global_id 4511 2024-03-29T11:07:13.102 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.111+0000 7f7e20df3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:07:13.102 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.111+0000 7f7e1affd640 1 -- 172.21.15.71:0/2120384431 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f7e14001a10 con 0x7f7e1c0b38c0 2024-03-29T11:07:13.102 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.111+0000 7f7e1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:13.102 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.111+0000 7f7e1affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:07:13.103 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.111+0000 7f7e1affd640 10 monclient: dump: 2024-03-29T11:07:13.103 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:13.103 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:13.103 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:13.103 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:13.103 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:13.103 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:13.103 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:13.103 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:13.103 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:13.103 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:13.103 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.111+0000 7f7e20df3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:07:13.103 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.111+0000 7f7e1affd640 1 -- 172.21.15.71:0/2120384431 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e14001c30 con 0x7f7e1c0b38c0 2024-03-29T11:07:13.103 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.111+0000 7f7e1affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:13.104 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.111+0000 7f7e20df3640 10 monclient: get_monmap_and_config success 2024-03-29T11:07:13.104 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.111+0000 7f7e20df3640 4 set_mon_vals no callback set 2024-03-29T11:07:13.104 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.112+0000 7f7e20df3640 10 monclient: shutdown 2024-03-29T11:07:13.104 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.112+0000 7f7e20df3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:13.104 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.112+0000 7f7e20df3640 1 -- 172.21.15.71:0/2120384431 >> v1:172.21.15.71:6790/0 conn(0x7f7e1c0b38c0 legacy=0x7f7e1c14d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:13.104 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.112+0000 7f7e20df3640 1 -- 172.21.15.71:0/2120384431 shutdown_connections 2024-03-29T11:07:13.104 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.112+0000 7f7e20df3640 1 -- 172.21.15.71:0/2120384431 >> 172.21.15.71:0/2120384431 conn(0x7f7e1c066a00 msgr2=0x7f7e1c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:13.104 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.113+0000 7f7e20df3640 1 -- 172.21.15.71:0/2120384431 shutdown_connections 2024-03-29T11:07:13.104 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.113+0000 7f7e20df3640 1 -- 172.21.15.71:0/2120384431 wait complete. 2024-03-29T11:07:13.104 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.113+0000 7f7e20df3640 10 monclient: build_initial_monmap 2024-03-29T11:07:13.104 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.114+0000 7f7e20df3640 10 monclient: monmap: 2024-03-29T11:07:13.104 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:07:13.104 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:07:13.104 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:07:13.105 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:07:13.105 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:13.105 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:13.105 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:07:13.105 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:07:13.105 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:07:13.105 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:13.105 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.114+0000 7f7e20df3640 1 librados: starting msgr at 2024-03-29T11:07:13.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.114+0000 7f7e20df3640 1 librados: starting objecter 2024-03-29T11:07:13.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.114+0000 7f7e20df3640 1 Processor -- start 2024-03-29T11:07:13.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.114+0000 7f7e20df3640 1 -- start start 2024-03-29T11:07:13.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.114+0000 7f7e20df3640 1 librados: setting wanted keys 2024-03-29T11:07:13.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.114+0000 7f7e20df3640 1 librados: calling monclient init 2024-03-29T11:07:13.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.114+0000 7f7e20df3640 10 monclient: init 2024-03-29T11:07:13.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.115+0000 7f7e20df3640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:13.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.115+0000 7f7e20df3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:07:13.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.115+0000 7f7e20df3640 10 monclient(hunting): picked mon.noname-b con 0x7f7e1c0a9c00 addr v1:172.21.15.137:6789/0 2024-03-29T11:07:13.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.115+0000 7f7e20df3640 10 monclient(hunting): picked mon.noname-a con 0x7f7e1c0b2800 addr v1:172.21.15.71:6789/0 2024-03-29T11:07:13.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.115+0000 7f7e20df3640 10 monclient(hunting): picked mon.noname-c con 0x7f7e1c0b38c0 addr v1:172.21.15.71:6790/0 2024-03-29T11:07:13.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.115+0000 7f7e20df3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e1c06ef10 con 0x7f7e1c0b2800 2024-03-29T11:07:13.106 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.115+0000 7f7e20df3640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e1c06f180 con 0x7f7e1c0a9c00 2024-03-29T11:07:13.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.115+0000 7f7e20df3640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e1c06f460 con 0x7f7e1c0b38c0 2024-03-29T11:07:13.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.115+0000 7f7e20df3640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:13.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.115+0000 7f7e20df3640 10 monclient(hunting): authenticate will time out at 1189.373570s 2024-03-29T11:07:13.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.117+0000 7f7e1bfff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f7e1c0b38c0 0x7f7e1c13d0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:43898/0 (socket says 172.21.15.71:43898) 2024-03-29T11:07:13.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.117+0000 7f7e1bfff640 1 -- 172.21.15.71:0/1399894245 learned_addr learned my addr 172.21.15.71:0/1399894245 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:07:13.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.117+0000 7f7e18ff9640 10 client.?.objecter ms_handle_connect 0x7f7e1c0b38c0 2024-03-29T11:07:13.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.117+0000 7f7e18ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:13.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.117+0000 7f7e18ff9640 1 -- 172.21.15.71:0/1399894245 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2253553040 0 0) 0x7f7e1c06f460 con 0x7f7e1c0b38c0 2024-03-29T11:07:13.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.118+0000 7f7e18ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:07:13.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.118+0000 7f7e18ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:13.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.118+0000 7f7e18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:13.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.118+0000 7f7e18ff9640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:07:13.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.118+0000 7f7e18ff9640 1 -- 172.21.15.71:0/1399894245 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dfc004eb0 con 0x7f7e1c0b38c0 2024-03-29T11:07:13.107 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.118+0000 7f7e18ff9640 1 -- 172.21.15.71:0/1399894245 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3375634642 0 0) 0x7f7dfc004eb0 con 0x7f7e1c0b38c0 2024-03-29T11:07:13.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.118+0000 7f7e18ff9640 1 -- 172.21.15.71:0/1399894245 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e1c06f460 con 0x7f7e1c0b38c0 2024-03-29T11:07:13.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.118+0000 7f7e18ff9640 1 -- 172.21.15.71:0/1399894245 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f7e14004d00 con 0x7f7e1c0b38c0 2024-03-29T11:07:13.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.118+0000 7f7e18ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:13.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.118+0000 7f7e18ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:07:13.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.119+0000 7f7e18ff9640 10 monclient(hunting): dump: 2024-03-29T11:07:13.108 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:13.108 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:13.108 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:13.108 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:13.108 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:13.108 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:13.108 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:13.108 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:13.109 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:13.109 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:13.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.119+0000 7f7e18ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:07:13.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.119+0000 7f7e18ff9640 10 client.?.objecter ms_handle_connect 0x7f7e1c0a9c00 2024-03-29T11:07:13.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.119+0000 7f7e18ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:13.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.119+0000 7f7e18ff9640 1 -- 172.21.15.71:0/1399894245 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3726992648 0 0) 0x7f7e1c06f460 con 0x7f7e1c0b38c0 2024-03-29T11:07:13.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.119+0000 7f7e18ff9640 1 -- 172.21.15.71:0/1399894245 >> v1:172.21.15.137:6789/0 conn(0x7f7e1c0a9c00 legacy=0x7f7e1c1db6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:13.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.119+0000 7f7e18ff9640 1 -- 172.21.15.71:0/1399894245 >> v1:172.21.15.71:6789/0 conn(0x7f7e1c0b2800 legacy=0x7f7e1c13c9a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:07:13.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.120+0000 7f7e18ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:07:13.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.120+0000 7f7e18ff9640 1 monclient: found mon.c 2024-03-29T11:07:13.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.120+0000 7f7e18ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:13.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.120+0000 7f7e18ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:07:13.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.120+0000 7f7e18ff9640 1 -- 172.21.15.71:0/1399894245 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1c13c480 con 0x7f7e1c0b38c0 2024-03-29T11:07:13.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.120+0000 7f7e18ff9640 10 monclient: _finish_auth 0 2024-03-29T11:07:13.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.120+0000 7f7e18ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:07:13.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.120+0000 7f7e18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:13.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.120+0000 7f7e20df3640 5 monclient: authenticate success, global_id 4514 2024-03-29T11:07:13.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.120+0000 7f7e20df3640 10 monclient: _renew_subs 2024-03-29T11:07:13.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.120+0000 7f7e20df3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:07:13.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.120+0000 7f7e20df3640 1 -- 172.21.15.71:0/1399894245 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e1c138540 con 0x7f7e1c0b38c0 2024-03-29T11:07:13.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.120+0000 7f7e20df3640 10 client.4514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:07:13.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.120+0000 7f7e20df3640 10 monclient: _renew_subs 2024-03-29T11:07:13.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.120+0000 7f7e20df3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:07:13.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.120+0000 7f7e20df3640 1 -- 172.21.15.71:0/1399894245 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e1c06e100 con 0x7f7e1c0b38c0 2024-03-29T11:07:13.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.120+0000 7f7e20df3640 1 librados: init done 2024-03-29T11:07:13.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.120+0000 7f7e18ff9640 1 -- 172.21.15.71:0/1399894245 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e14005200 con 0x7f7e1c0b38c0 2024-03-29T11:07:13.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.121+0000 7f7e18ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:13.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.121+0000 7f7e18ff9640 1 -- 172.21.15.71:0/1399894245 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f7e14005590 con 0x7f7e1c0b38c0 2024-03-29T11:07:13.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.121+0000 7f7e18ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:13.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.121+0000 7f7e18ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:07:13.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.121+0000 7f7e18ff9640 10 monclient: dump: 2024-03-29T11:07:13.111 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:13.112 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:13.112 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:13.112 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:13.112 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:13.112 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:13.112 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:13.112 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:13.112 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:13.112 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:13.113 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.121+0000 7f7e1a7fc640 4 set_mon_vals no callback set 2024-03-29T11:07:13.113 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.123+0000 7f7e18ff9640 1 -- 172.21.15.71:0/1399894245 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f7e14001e20 con 0x7f7e1c0b38c0 2024-03-29T11:07:13.113 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.123+0000 7f7e20df3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:13.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.123+0000 7f7e20df3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:13.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.123+0000 7f7e20df3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:07:13.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.123+0000 7f7e20df3640 1 -- 172.21.15.71:0/1399894245 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e00005080 con 0x7f7e1c0b38c0 2024-03-29T11:07:13.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.123+0000 7f7e18ff9640 1 -- 172.21.15.71:0/1399894245 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f7e140850e0 con 0x7f7e1c0b38c0 2024-03-29T11:07:13.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.123+0000 7f7e18ff9640 10 client.4514.objecter ms_dispatch 0x7f7e1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:07:13.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.123+0000 7f7e18ff9640 3 client.4514.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:07:13.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.123+0000 7f7e18ff9640 3 client.4514.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:07:13.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.123+0000 7f7e18ff9640 20 client.4514.objecter dump_active .. 0 homeless 2024-03-29T11:07:13.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.124+0000 7f7e1b7fe640 10 monclient: get_auth_request con 0x7f7dfc06dbb0 auth_method 0 2024-03-29T11:07:13.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.125+0000 7f7e18ff9640 10 client.4514.objecter ms_handle_connect 0x7f7dfc06dbb0 2024-03-29T11:07:13.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.132+0000 7f7e18ff9640 1 -- 172.21.15.71:0/1399894245 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7e140509c0 con 0x7f7e1c0b38c0 2024-03-29T11:07:13.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.132+0000 7f7e18ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:13.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.132+0000 7f7e18ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:07:13.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.455+0000 7f7e20df3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:13.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.455+0000 7f7e20df3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:13.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.455+0000 7f7e20df3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:07:13.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.455+0000 7f7e20df3640 1 -- 172.21.15.71:0/1399894245 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e00005330 con 0x7f7e1c0b38c0 2024-03-29T11:07:13.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.458+0000 7f7e18ff9640 1 -- 172.21.15.71:0/1399894245 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f7e140528f0 con 0x7f7e1c0b38c0 2024-03-29T11:07:13.437 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.458+0000 7f7e18ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:13.437 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.458+0000 7f7e18ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:07:13.437 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:07:13.437 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:07:13.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.460+0000 7f7e20df3640 10 librados: watch_flush enter 2024-03-29T11:07:13.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.460+0000 7f7e20df3640 10 librados: watch_flush exit 2024-03-29T11:07:13.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.461+0000 7f7e20df3640 20 client.4514.objecter shutdown clearing up homeless session... 2024-03-29T11:07:13.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.461+0000 7f7e20df3640 10 client.4514.objecter successfully canceled tick 2024-03-29T11:07:13.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.463+0000 7f7e20df3640 1 -- 172.21.15.71:0/1399894245 >> v1:172.21.15.137:6816/973332665 conn(0x7f7dfc06dbb0 legacy=0x7f7dfc070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:13.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.463+0000 7f7e20df3640 10 monclient: shutdown 2024-03-29T11:07:13.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.463+0000 7f7e20df3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:13.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.463+0000 7f7e20df3640 1 -- 172.21.15.71:0/1399894245 >> v1:172.21.15.71:6790/0 conn(0x7f7e1c0b38c0 legacy=0x7f7e1c13d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:13.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.463+0000 7f7e20df3640 1 -- 172.21.15.71:0/1399894245 shutdown_connections 2024-03-29T11:07:13.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.463+0000 7f7e20df3640 1 -- 172.21.15.71:0/1399894245 >> 172.21.15.71:0/1399894245 conn(0x7f7e1c066a00 msgr2=0x7f7e1c140450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:13.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.463+0000 7f7e20df3640 1 -- 172.21.15.71:0/1399894245 shutdown_connections 2024-03-29T11:07:13.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.463+0000 7f7e20df3640 1 -- 172.21.15.71:0/1399894245 wait complete. 2024-03-29T11:07:13.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:13.463+0000 7f7e20df3640 1 librados: shutdown 2024-03-29T11:07:13.459 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:07:20.267 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:07:20.400 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.421+0000 7fd93e47d640 10 monclient: get_monmap_and_config 2024-03-29T11:07:20.400 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.421+0000 7fd93e47d640 10 monclient: build_initial_monmap 2024-03-29T11:07:20.400 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.421+0000 7fd93e47d640 10 monclient: monmap: 2024-03-29T11:07:20.400 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:07:20.400 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:20.400 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:07:20.421985+0000 2024-03-29T11:07:20.400 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:07:20.421985+0000 2024-03-29T11:07:20.400 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:20.400 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:20.400 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:07:20.400 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:07:20.401 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:07:20.401 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:20.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.422+0000 7fd93e47d640 1 Processor -- start 2024-03-29T11:07:20.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.422+0000 7fd93e47d640 1 -- start start 2024-03-29T11:07:20.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.422+0000 7fd93e47d640 10 monclient: init 2024-03-29T11:07:20.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.423+0000 7fd93e47d640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:20.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.423+0000 7fd93e47d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:07:20.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.423+0000 7fd93e47d640 10 monclient(hunting): picked mon.noname-c con 0x7fd9380b0860 addr v1:172.21.15.137:6789/0 2024-03-29T11:07:20.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.423+0000 7fd9377fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fd93813ef30 0x7fd9380af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:58562/0 (socket says 172.21.15.71:58562) 2024-03-29T11:07:20.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.423+0000 7fd9377fe640 1 -- 172.21.15.71:0/1202721204 learned_addr learned my addr 172.21.15.71:0/1202721204 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:07:20.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.423+0000 7fd93e47d640 10 monclient(hunting): picked mon.noname-b con 0x7fd93813ef30 addr v1:172.21.15.71:6790/0 2024-03-29T11:07:20.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.423+0000 7fd93e47d640 10 monclient(hunting): picked mon.noname-a con 0x7fd9380af8e0 addr v1:172.21.15.71:6789/0 2024-03-29T11:07:20.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.424+0000 7fd93e47d640 1 -- 172.21.15.71:0/1202721204 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9380b0c40 con 0x7fd9380af8e0 2024-03-29T11:07:20.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.424+0000 7fd93e47d640 1 -- 172.21.15.71:0/1202721204 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9380aed10 con 0x7fd9380b0860 2024-03-29T11:07:20.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.424+0000 7fd93e47d640 1 -- 172.21.15.71:0/1202721204 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9380afd50 con 0x7fd93813ef30 2024-03-29T11:07:20.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.424+0000 7fd93e47d640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:20.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.424+0000 7fd93e47d640 10 monclient(hunting): authenticate will time out at 1196.682443s 2024-03-29T11:07:20.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.424+0000 7fd936ffd640 1 -- 172.21.15.71:0/1202721204 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211868464 0 0) 0x7fd9380afd50 con 0x7fd93813ef30 2024-03-29T11:07:20.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.424+0000 7fd936ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:07:20.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.424+0000 7fd936ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:20.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.424+0000 7fd936ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:20.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.424+0000 7fd936ffd640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:07:20.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.424+0000 7fd936ffd640 1 -- 172.21.15.71:0/1202721204 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd920003520 con 0x7fd93813ef30 2024-03-29T11:07:20.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.424+0000 7fd936ffd640 1 -- 172.21.15.71:0/1202721204 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 311966808 0 0) 0x7fd9380aed10 con 0x7fd9380b0860 2024-03-29T11:07:20.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.424+0000 7fd936ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:07:20.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.424+0000 7fd936ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:20.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.424+0000 7fd936ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:20.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.424+0000 7fd936ffd640 10 monclient(hunting): my global_id is 4435 2024-03-29T11:07:20.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.424+0000 7fd936ffd640 1 -- 172.21.15.71:0/1202721204 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9380afd50 con 0x7fd9380b0860 2024-03-29T11:07:20.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.426+0000 7fd936ffd640 1 -- 172.21.15.71:0/1202721204 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4263005496 0 0) 0x7fd920003520 con 0x7fd93813ef30 2024-03-29T11:07:20.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.426+0000 7fd936ffd640 1 -- 172.21.15.71:0/1202721204 >> v1:172.21.15.137:6789/0 conn(0x7fd9380b0860 legacy=0x7fd93813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:20.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.426+0000 7fd936ffd640 1 -- 172.21.15.71:0/1202721204 >> v1:172.21.15.71:6789/0 conn(0x7fd9380af8e0 legacy=0x7fd938149a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:07:20.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.427+0000 7fd936ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:07:20.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.427+0000 7fd936ffd640 1 monclient: found mon.noname-b 2024-03-29T11:07:20.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.427+0000 7fd936ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:20.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.427+0000 7fd936ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:07:20.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.427+0000 7fd936ffd640 1 -- 172.21.15.71:0/1202721204 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9380ab1b0 con 0x7fd93813ef30 2024-03-29T11:07:20.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.427+0000 7fd936ffd640 10 monclient: _finish_auth 0 2024-03-29T11:07:20.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.427+0000 7fd936ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:07:20.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.427+0000 7fd936ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:20.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.427+0000 7fd936ffd640 1 -- 172.21.15.71:0/1202721204 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fd928001690 con 0x7fd93813ef30 2024-03-29T11:07:20.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.427+0000 7fd936ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:20.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.427+0000 7fd936ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:07:20.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.427+0000 7fd936ffd640 10 monclient: dump: 2024-03-29T11:07:20.408 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:20.408 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:20.408 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:20.408 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:20.408 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:20.408 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:20.409 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:20.409 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:20.409 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:20.409 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:20.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.427+0000 7fd936ffd640 1 -- 172.21.15.71:0/1202721204 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9280018b0 con 0x7fd93813ef30 2024-03-29T11:07:20.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.427+0000 7fd936ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:20.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.427+0000 7fd936ffd640 1 -- 172.21.15.71:0/1202721204 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fd928001c40 con 0x7fd93813ef30 2024-03-29T11:07:20.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.427+0000 7fd936ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:20.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.427+0000 7fd936ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:07:20.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.427+0000 7fd936ffd640 10 monclient: dump: 2024-03-29T11:07:20.409 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:20.409 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:20.409 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:20.409 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:20.409 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:20.410 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:20.410 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:20.410 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:20.410 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:20.410 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:20.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.427+0000 7fd93e47d640 5 monclient: authenticate success, global_id 4517 2024-03-29T11:07:20.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.427+0000 7fd93e47d640 10 monclient: get_monmap_and_config success 2024-03-29T11:07:20.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.427+0000 7fd93e47d640 4 set_mon_vals no callback set 2024-03-29T11:07:20.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.428+0000 7fd93e47d640 10 monclient: shutdown 2024-03-29T11:07:20.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.428+0000 7fd93e47d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:20.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.428+0000 7fd93e47d640 1 -- 172.21.15.71:0/1202721204 >> v1:172.21.15.71:6790/0 conn(0x7fd93813ef30 legacy=0x7fd9380af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:20.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.429+0000 7fd93e47d640 1 -- 172.21.15.71:0/1202721204 shutdown_connections 2024-03-29T11:07:20.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.429+0000 7fd93e47d640 1 -- 172.21.15.71:0/1202721204 >> 172.21.15.71:0/1202721204 conn(0x7fd93806b660 msgr2=0x7fd93806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:20.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.429+0000 7fd93e47d640 1 -- 172.21.15.71:0/1202721204 shutdown_connections 2024-03-29T11:07:20.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.429+0000 7fd93e47d640 1 -- 172.21.15.71:0/1202721204 wait complete. 2024-03-29T11:07:20.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.430+0000 7fd93e47d640 10 monclient: build_initial_monmap 2024-03-29T11:07:20.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.430+0000 7fd93e47d640 10 monclient: monmap: 2024-03-29T11:07:20.415 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:07:20.415 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:07:20.415 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:07:20.415 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:07:20.415 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:20.415 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:20.415 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:07:20.416 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:07:20.416 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:07:20.416 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:20.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.430+0000 7fd93e47d640 1 librados: starting msgr at 2024-03-29T11:07:20.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.431+0000 7fd93e47d640 1 librados: starting objecter 2024-03-29T11:07:20.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.431+0000 7fd93e47d640 1 Processor -- start 2024-03-29T11:07:20.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.431+0000 7fd93e47d640 1 -- start start 2024-03-29T11:07:20.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.431+0000 7fd93e47d640 1 librados: setting wanted keys 2024-03-29T11:07:20.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.431+0000 7fd93e47d640 1 librados: calling monclient init 2024-03-29T11:07:20.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.431+0000 7fd93e47d640 10 monclient: init 2024-03-29T11:07:20.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.432+0000 7fd93e47d640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:20.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.432+0000 7fd93e47d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:07:20.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.433+0000 7fd93e47d640 10 monclient(hunting): picked mon.noname-a con 0x7fd9380af8e0 addr v1:172.21.15.71:6789/0 2024-03-29T11:07:20.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.433+0000 7fd93e47d640 10 monclient(hunting): picked mon.noname-c con 0x7fd9380b0860 addr v1:172.21.15.71:6790/0 2024-03-29T11:07:20.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.433+0000 7fd93e47d640 10 monclient(hunting): picked mon.noname-b con 0x7fd93813ef30 addr v1:172.21.15.137:6789/0 2024-03-29T11:07:20.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.433+0000 7fd93e47d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9381d6a20 con 0x7fd9380af8e0 2024-03-29T11:07:20.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.433+0000 7fd93e47d640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9381c50b0 con 0x7fd93813ef30 2024-03-29T11:07:20.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.433+0000 7fd93e47d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd938135170 con 0x7fd9380b0860 2024-03-29T11:07:20.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.433+0000 7fd93e47d640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:20.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.433+0000 7fd93e47d640 10 monclient(hunting): authenticate will time out at 1196.691398s 2024-03-29T11:07:20.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.433+0000 7fd9377fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fd9380b0860 0x7fd938134350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:37010/0 (socket says 172.21.15.71:37010) 2024-03-29T11:07:20.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.433+0000 7fd9377fe640 1 -- 172.21.15.71:0/4262236466 learned_addr learned my addr 172.21.15.71:0/4262236466 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:07:20.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.433+0000 7fd934ff9640 10 client.?.objecter ms_handle_connect 0x7fd9380b0860 2024-03-29T11:07:20.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.433+0000 7fd934ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:20.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.433+0000 7fd934ff9640 1 -- 172.21.15.71:0/4262236466 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434812639 0 0) 0x7fd938135170 con 0x7fd9380b0860 2024-03-29T11:07:20.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.433+0000 7fd934ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:07:20.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.433+0000 7fd934ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:20.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.433+0000 7fd934ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:20.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.433+0000 7fd934ff9640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:07:20.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.433+0000 7fd934ff9640 1 -- 172.21.15.71:0/4262236466 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd924004eb0 con 0x7fd9380b0860 2024-03-29T11:07:20.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd934ff9640 10 client.?.objecter ms_handle_connect 0x7fd93813ef30 2024-03-29T11:07:20.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd934ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:20.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd934ff9640 1 -- 172.21.15.71:0/4262236466 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3430662510 0 0) 0x7fd924004eb0 con 0x7fd9380b0860 2024-03-29T11:07:20.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd934ff9640 1 -- 172.21.15.71:0/4262236466 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd938135170 con 0x7fd9380b0860 2024-03-29T11:07:20.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd934ff9640 1 -- 172.21.15.71:0/4262236466 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fd928001bc0 con 0x7fd9380b0860 2024-03-29T11:07:20.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd934ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:20.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd934ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:07:20.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd934ff9640 10 monclient(hunting): dump: 2024-03-29T11:07:20.418 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:20.418 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:20.418 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:20.418 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:20.418 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:20.419 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:20.419 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:20.419 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:20.419 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:20.419 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:20.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd934ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:07:20.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd934ff9640 1 -- 172.21.15.71:0/4262236466 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890700186 0 0) 0x7fd9381c50b0 con 0x7fd93813ef30 2024-03-29T11:07:20.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd934ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:07:20.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd934ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:20.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd934ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:20.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd934ff9640 10 monclient(hunting): my global_id is 4438 2024-03-29T11:07:20.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd934ff9640 1 -- 172.21.15.71:0/4262236466 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd924004eb0 con 0x7fd93813ef30 2024-03-29T11:07:20.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd934ff9640 1 -- 172.21.15.71:0/4262236466 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3242311341 0 0) 0x7fd938135170 con 0x7fd9380b0860 2024-03-29T11:07:20.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd934ff9640 1 -- 172.21.15.71:0/4262236466 >> v1:172.21.15.137:6789/0 conn(0x7fd93813ef30 legacy=0x7fd938134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:20.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd934ff9640 1 -- 172.21.15.71:0/4262236466 >> v1:172.21.15.71:6789/0 conn(0x7fd9380af8e0 legacy=0x7fd9381d2d30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:07:20.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd934ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:07:20.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd934ff9640 1 monclient: found mon.c 2024-03-29T11:07:20.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd934ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:20.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd934ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:07:20.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd934ff9640 1 -- 172.21.15.71:0/4262236466 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9381c81c0 con 0x7fd9380b0860 2024-03-29T11:07:20.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd934ff9640 10 monclient: _finish_auth 0 2024-03-29T11:07:20.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd934ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:07:20.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd934ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:20.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd93e47d640 5 monclient: authenticate success, global_id 4520 2024-03-29T11:07:20.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd93e47d640 10 monclient: _renew_subs 2024-03-29T11:07:20.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd93e47d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:07:20.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.434+0000 7fd93e47d640 1 -- 172.21.15.71:0/4262236466 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9381d3440 con 0x7fd9380b0860 2024-03-29T11:07:20.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.435+0000 7fd93e47d640 10 client.4520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:07:20.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.435+0000 7fd93e47d640 10 monclient: _renew_subs 2024-03-29T11:07:20.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.435+0000 7fd93e47d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:07:20.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.435+0000 7fd93e47d640 1 -- 172.21.15.71:0/4262236466 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9381d7090 con 0x7fd9380b0860 2024-03-29T11:07:20.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.435+0000 7fd93e47d640 1 librados: init done 2024-03-29T11:07:20.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.435+0000 7fd934ff9640 1 -- 172.21.15.71:0/4262236466 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd928004a90 con 0x7fd9380b0860 2024-03-29T11:07:20.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.435+0000 7fd934ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:20.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.435+0000 7fd934ff9640 1 -- 172.21.15.71:0/4262236466 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fd928004da0 con 0x7fd9380b0860 2024-03-29T11:07:20.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.435+0000 7fd934ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:20.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.435+0000 7fd934ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:07:20.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.435+0000 7fd934ff9640 10 monclient: dump: 2024-03-29T11:07:20.421 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:20.421 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:20.421 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:20.421 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:20.421 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:20.421 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:20.422 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:20.422 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:20.422 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:20.422 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:20.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.435+0000 7fd9367fc640 4 set_mon_vals no callback set 2024-03-29T11:07:20.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.437+0000 7fd934ff9640 1 -- 172.21.15.71:0/4262236466 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7fd92801ad70 con 0x7fd9380b0860 2024-03-29T11:07:20.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.437+0000 7fd93e47d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:20.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.437+0000 7fd93e47d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:20.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.437+0000 7fd93e47d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:07:20.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.437+0000 7fd93e47d640 1 -- 172.21.15.71:0/4262236466 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd938075f10 con 0x7fd9380b0860 2024-03-29T11:07:20.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.438+0000 7fd934ff9640 1 -- 172.21.15.71:0/4262236466 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7fd928084b60 con 0x7fd9380b0860 2024-03-29T11:07:20.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.438+0000 7fd934ff9640 10 client.4520.objecter ms_dispatch 0x7fd938000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:07:20.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.438+0000 7fd934ff9640 3 client.4520.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:07:20.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.438+0000 7fd934ff9640 3 client.4520.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:07:20.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.438+0000 7fd934ff9640 20 client.4520.objecter dump_active .. 0 homeless 2024-03-29T11:07:20.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.438+0000 7fd937fff640 10 monclient: get_auth_request con 0x7fd92406dd70 auth_method 0 2024-03-29T11:07:20.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.438+0000 7fd934ff9640 10 client.4520.objecter ms_handle_connect 0x7fd92406dd70 2024-03-29T11:07:20.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.444+0000 7fd934ff9640 1 -- 172.21.15.71:0/4262236466 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd928050440 con 0x7fd9380b0860 2024-03-29T11:07:20.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.444+0000 7fd934ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:20.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.444+0000 7fd934ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:07:20.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.754+0000 7fd93e47d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:20.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.754+0000 7fd93e47d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:20.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.754+0000 7fd93e47d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:07:20.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.754+0000 7fd93e47d640 1 -- 172.21.15.71:0/4262236466 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd938072330 con 0x7fd9380b0860 2024-03-29T11:07:20.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.755+0000 7fd934ff9640 1 -- 172.21.15.71:0/4262236466 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7fd928052370 con 0x7fd9380b0860 2024-03-29T11:07:20.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.755+0000 7fd934ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:20.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.755+0000 7fd934ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:07:20.734 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:07:20.734 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:07:20.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.758+0000 7fd93e47d640 10 librados: watch_flush enter 2024-03-29T11:07:20.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.758+0000 7fd93e47d640 10 librados: watch_flush exit 2024-03-29T11:07:20.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.758+0000 7fd93e47d640 20 client.4520.objecter shutdown clearing up homeless session... 2024-03-29T11:07:20.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.758+0000 7fd93e47d640 10 client.4520.objecter successfully canceled tick 2024-03-29T11:07:20.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.758+0000 7fd93e47d640 1 -- 172.21.15.71:0/4262236466 >> v1:172.21.15.137:6816/973332665 conn(0x7fd92406dd70 legacy=0x7fd924070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:20.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.758+0000 7fd93e47d640 10 monclient: shutdown 2024-03-29T11:07:20.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.758+0000 7fd93e47d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:20.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.758+0000 7fd93e47d640 1 -- 172.21.15.71:0/4262236466 >> v1:172.21.15.71:6790/0 conn(0x7fd9380b0860 legacy=0x7fd938134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:20.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.758+0000 7fd93e47d640 1 -- 172.21.15.71:0/4262236466 shutdown_connections 2024-03-29T11:07:20.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.758+0000 7fd93e47d640 1 -- 172.21.15.71:0/4262236466 >> 172.21.15.71:0/4262236466 conn(0x7fd93806b660 msgr2=0x7fd93813e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:20.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.758+0000 7fd93e47d640 1 -- 172.21.15.71:0/4262236466 shutdown_connections 2024-03-29T11:07:20.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.758+0000 7fd93e47d640 1 -- 172.21.15.71:0/4262236466 wait complete. 2024-03-29T11:07:20.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:20.758+0000 7fd93e47d640 1 librados: shutdown 2024-03-29T11:07:20.752 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:07:27.559 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:07:27.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.729+0000 7fdde2c0f640 10 monclient: get_monmap_and_config 2024-03-29T11:07:27.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.729+0000 7fdde2c0f640 10 monclient: build_initial_monmap 2024-03-29T11:07:27.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.729+0000 7fdde2c0f640 10 monclient: monmap: 2024-03-29T11:07:27.710 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:07:27.710 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:27.710 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:07:27.730286+0000 2024-03-29T11:07:27.710 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:07:27.730286+0000 2024-03-29T11:07:27.710 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:27.710 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:27.710 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:07:27.710 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:07:27.710 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:07:27.710 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:27.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.730+0000 7fdde2c0f640 1 Processor -- start 2024-03-29T11:07:27.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.730+0000 7fdde2c0f640 1 -- start start 2024-03-29T11:07:27.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.730+0000 7fdde2c0f640 10 monclient: init 2024-03-29T11:07:27.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.731+0000 7fdde2c0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:27.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.731+0000 7fdde2c0f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:07:27.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.731+0000 7fdde2c0f640 10 monclient(hunting): picked mon.noname-a con 0x7fdddc0a9930 addr v1:172.21.15.71:6789/0 2024-03-29T11:07:27.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.731+0000 7fdde2c0f640 10 monclient(hunting): picked mon.noname-c con 0x7fdddc149dc0 addr v1:172.21.15.137:6789/0 2024-03-29T11:07:27.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.732+0000 7fdde2c0f640 10 monclient(hunting): picked mon.noname-b con 0x7fdddc14ff40 addr v1:172.21.15.71:6790/0 2024-03-29T11:07:27.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.732+0000 7fdde2c0f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdddc0aae60 con 0x7fdddc0a9930 2024-03-29T11:07:27.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.732+0000 7fdde2c0f640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdddc0b4310 con 0x7fdddc149dc0 2024-03-29T11:07:27.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.732+0000 7fdde2c0f640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdddc0afdd0 con 0x7fdddc14ff40 2024-03-29T11:07:27.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.732+0000 7fdde2c0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:27.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.732+0000 7fdde2c0f640 10 monclient(hunting): authenticate will time out at 1203.990220s 2024-03-29T11:07:27.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.732+0000 7fdde11ac640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fdddc14ff40 0x7fdddc152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:56656/0 (socket says 172.21.15.71:56656) 2024-03-29T11:07:27.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.732+0000 7fdde11ac640 1 -- 172.21.15.71:0/221757508 learned_addr learned my addr 172.21.15.71:0/221757508 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:07:27.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.732+0000 7fddd37fe640 1 -- 172.21.15.71:0/221757508 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3615034975 0 0) 0x7fdddc0afdd0 con 0x7fdddc14ff40 2024-03-29T11:07:27.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.732+0000 7fddd37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:07:27.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.732+0000 7fddd37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:27.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.732+0000 7fddd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:27.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.732+0000 7fddd37fe640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:07:27.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.732+0000 7fddd37fe640 1 -- 172.21.15.71:0/221757508 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddc4003520 con 0x7fdddc14ff40 2024-03-29T11:07:27.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.734+0000 7fddd37fe640 1 -- 172.21.15.71:0/221757508 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3287024573 0 0) 0x7fddc4003520 con 0x7fdddc14ff40 2024-03-29T11:07:27.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.734+0000 7fddd37fe640 1 -- 172.21.15.71:0/221757508 >> v1:172.21.15.137:6789/0 conn(0x7fdddc149dc0 legacy=0x7fdddc14a240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:07:27.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.738+0000 7fddd37fe640 1 -- 172.21.15.71:0/221757508 >> v1:172.21.15.71:6789/0 conn(0x7fdddc0a9930 legacy=0x7fdddc143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:27.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.738+0000 7fddd37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:07:27.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.738+0000 7fddd37fe640 1 monclient: found mon.noname-b 2024-03-29T11:07:27.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.738+0000 7fddd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:27.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.738+0000 7fddd37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:07:27.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.738+0000 7fddd37fe640 1 -- 172.21.15.71:0/221757508 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc14a950 con 0x7fdddc14ff40 2024-03-29T11:07:27.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.738+0000 7fddd37fe640 10 monclient: _finish_auth 0 2024-03-29T11:07:27.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.738+0000 7fddd37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:07:27.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.738+0000 7fddd37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:27.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.738+0000 7fddd37fe640 1 -- 172.21.15.71:0/221757508 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fddd4001a10 con 0x7fdddc14ff40 2024-03-29T11:07:27.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.738+0000 7fddd37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:27.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.738+0000 7fddd37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:07:27.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.738+0000 7fddd37fe640 10 monclient: dump: 2024-03-29T11:07:27.718 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:27.719 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:27.719 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:27.719 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:27.719 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:27.719 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:27.719 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:27.719 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:27.719 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:27.719 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:27.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.738+0000 7fdde2c0f640 5 monclient: authenticate success, global_id 4523 2024-03-29T11:07:27.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.738+0000 7fdde2c0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:07:27.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.738+0000 7fddd37fe640 1 -- 172.21.15.71:0/221757508 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddd4001d20 con 0x7fdddc14ff40 2024-03-29T11:07:27.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.738+0000 7fddd37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:27.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.738+0000 7fddd37fe640 1 -- 172.21.15.71:0/221757508 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fddd40020b0 con 0x7fdddc14ff40 2024-03-29T11:07:27.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.738+0000 7fddd37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:27.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.739+0000 7fddd37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:07:27.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.739+0000 7fddd37fe640 10 monclient: dump: 2024-03-29T11:07:27.720 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:27.720 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:27.720 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:27.720 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:27.720 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:27.720 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:27.720 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:27.720 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:27.720 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:27.720 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:27.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.739+0000 7fdde2c0f640 10 monclient: get_monmap_and_config success 2024-03-29T11:07:27.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.739+0000 7fdde2c0f640 4 set_mon_vals no callback set 2024-03-29T11:07:27.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.739+0000 7fdde2c0f640 10 monclient: shutdown 2024-03-29T11:07:27.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.739+0000 7fdde2c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:27.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.739+0000 7fdde2c0f640 1 -- 172.21.15.71:0/221757508 >> v1:172.21.15.71:6790/0 conn(0x7fdddc14ff40 legacy=0x7fdddc152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:27.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.740+0000 7fdde2c0f640 1 -- 172.21.15.71:0/221757508 shutdown_connections 2024-03-29T11:07:27.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.740+0000 7fdde2c0f640 1 -- 172.21.15.71:0/221757508 >> 172.21.15.71:0/221757508 conn(0x7fdddc0b35e0 msgr2=0x7fdddc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:27.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.740+0000 7fdde2c0f640 1 -- 172.21.15.71:0/221757508 shutdown_connections 2024-03-29T11:07:27.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.740+0000 7fdde2c0f640 1 -- 172.21.15.71:0/221757508 wait complete. 2024-03-29T11:07:27.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.741+0000 7fdde2c0f640 10 monclient: build_initial_monmap 2024-03-29T11:07:27.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.741+0000 7fdde2c0f640 10 monclient: monmap: 2024-03-29T11:07:27.722 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:07:27.722 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:07:27.722 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:07:27.722 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:07:27.723 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:27.723 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:27.723 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:07:27.723 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:07:27.723 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:07:27.723 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:27.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.742+0000 7fdde2c0f640 1 librados: starting msgr at 2024-03-29T11:07:27.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.742+0000 7fdde2c0f640 1 librados: starting objecter 2024-03-29T11:07:27.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.742+0000 7fdde2c0f640 1 Processor -- start 2024-03-29T11:07:27.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.742+0000 7fdde2c0f640 1 -- start start 2024-03-29T11:07:27.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.742+0000 7fdde2c0f640 1 librados: setting wanted keys 2024-03-29T11:07:27.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.742+0000 7fdde2c0f640 1 librados: calling monclient init 2024-03-29T11:07:27.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.742+0000 7fdde2c0f640 10 monclient: init 2024-03-29T11:07:27.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.743+0000 7fdde2c0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:27.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.743+0000 7fdde2c0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:07:27.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.743+0000 7fdde2c0f640 10 monclient(hunting): picked mon.noname-b con 0x7fdddc0a9930 addr v1:172.21.15.137:6789/0 2024-03-29T11:07:27.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.743+0000 7fdde2c0f640 10 monclient(hunting): picked mon.noname-c con 0x7fdddc149dc0 addr v1:172.21.15.71:6790/0 2024-03-29T11:07:27.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.744+0000 7fdde2c0f640 10 monclient(hunting): picked mon.noname-a con 0x7fdddc14ff40 addr v1:172.21.15.71:6789/0 2024-03-29T11:07:27.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.744+0000 7fdde2c0f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdddc1db380 con 0x7fdddc14ff40 2024-03-29T11:07:27.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.744+0000 7fdde2c0f640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdddc0acb40 con 0x7fdddc0a9930 2024-03-29T11:07:27.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.744+0000 7fdde2c0f640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdddc0accd0 con 0x7fdddc149dc0 2024-03-29T11:07:27.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.744+0000 7fdde2c0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:27.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.744+0000 7fdde2c0f640 10 monclient(hunting): authenticate will time out at 1204.002371s 2024-03-29T11:07:27.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.744+0000 7fddd3fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fdddc149dc0 0x7fdddc1dbc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:34510/0 (socket says 172.21.15.71:34510) 2024-03-29T11:07:27.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.744+0000 7fddd3fff640 1 -- 172.21.15.71:0/1033067971 learned_addr learned my addr 172.21.15.71:0/1033067971 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:07:27.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.746+0000 7fddd17fa640 10 client.?.objecter ms_handle_connect 0x7fdddc149dc0 2024-03-29T11:07:27.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.746+0000 7fddd17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:27.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.746+0000 7fddd17fa640 1 -- 172.21.15.71:0/1033067971 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2839804781 0 0) 0x7fdddc0accd0 con 0x7fdddc149dc0 2024-03-29T11:07:27.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.746+0000 7fddd17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:07:27.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.746+0000 7fddd17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:27.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.746+0000 7fddd17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:27.726 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.746+0000 7fddd17fa640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:07:27.726 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.746+0000 7fddd17fa640 1 -- 172.21.15.71:0/1033067971 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddc8004eb0 con 0x7fdddc149dc0 2024-03-29T11:07:27.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.747+0000 7fddd17fa640 1 -- 172.21.15.71:0/1033067971 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1525046921 0 0) 0x7fddc8004eb0 con 0x7fdddc149dc0 2024-03-29T11:07:27.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.747+0000 7fddd17fa640 1 -- 172.21.15.71:0/1033067971 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdddc0accd0 con 0x7fdddc149dc0 2024-03-29T11:07:27.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.747+0000 7fddd17fa640 1 -- 172.21.15.71:0/1033067971 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fddcc001870 con 0x7fdddc149dc0 2024-03-29T11:07:27.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.747+0000 7fddd17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:27.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.747+0000 7fddd17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:07:27.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.747+0000 7fddd17fa640 10 monclient(hunting): dump: 2024-03-29T11:07:27.728 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:27.728 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:27.728 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:27.728 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:27.728 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:27.728 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:27.728 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:27.728 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:27.728 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:27.728 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:27.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.747+0000 7fddd17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:07:27.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.747+0000 7fddd17fa640 1 -- 172.21.15.71:0/1033067971 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1012338555 0 0) 0x7fdddc0accd0 con 0x7fdddc149dc0 2024-03-29T11:07:27.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.747+0000 7fddd17fa640 1 -- 172.21.15.71:0/1033067971 >> v1:172.21.15.137:6789/0 conn(0x7fdddc0a9930 legacy=0x7fdddc148810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:07:27.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.748+0000 7fddd17fa640 1 -- 172.21.15.71:0/1033067971 >> v1:172.21.15.71:6789/0 conn(0x7fdddc14ff40 legacy=0x7fdddc0ac430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:27.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.748+0000 7fddd17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:07:27.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.748+0000 7fddd17fa640 1 monclient: found mon.c 2024-03-29T11:07:27.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.748+0000 7fddd17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:27.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.748+0000 7fddd17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:07:27.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.748+0000 7fddd17fa640 1 -- 172.21.15.71:0/1033067971 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc0ab900 con 0x7fdddc149dc0 2024-03-29T11:07:27.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.748+0000 7fddd17fa640 10 monclient: _finish_auth 0 2024-03-29T11:07:27.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.748+0000 7fddd17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:07:27.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.748+0000 7fddd17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:27.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.748+0000 7fddd17fa640 1 -- 172.21.15.71:0/1033067971 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddcc004900 con 0x7fdddc149dc0 2024-03-29T11:07:27.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.748+0000 7fddd17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:27.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.748+0000 7fddd17fa640 1 -- 172.21.15.71:0/1033067971 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fddcc001870 con 0x7fdddc149dc0 2024-03-29T11:07:27.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.748+0000 7fddd17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:27.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.748+0000 7fddd17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:07:27.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.748+0000 7fddd17fa640 10 monclient: dump: 2024-03-29T11:07:27.730 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:27.730 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:27.730 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:27.730 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:27.730 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:27.730 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:27.730 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:27.731 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:27.731 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:27.731 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:27.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.748+0000 7fddd2ffd640 4 set_mon_vals no callback set 2024-03-29T11:07:27.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.750+0000 7fdde2c0f640 5 monclient: authenticate success, global_id 4526 2024-03-29T11:07:27.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.750+0000 7fdde2c0f640 10 monclient: _renew_subs 2024-03-29T11:07:27.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.750+0000 7fdde2c0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:07:27.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.750+0000 7fdde2c0f640 1 -- 172.21.15.71:0/1033067971 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdddc0abbb0 con 0x7fdddc149dc0 2024-03-29T11:07:27.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.750+0000 7fdde2c0f640 10 client.4526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:07:27.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.750+0000 7fdde2c0f640 10 monclient: _renew_subs 2024-03-29T11:07:27.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.750+0000 7fdde2c0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:07:27.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.750+0000 7fdde2c0f640 1 -- 172.21.15.71:0/1033067971 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdddc149360 con 0x7fdddc149dc0 2024-03-29T11:07:27.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.750+0000 7fdde2c0f640 1 librados: init done 2024-03-29T11:07:27.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.752+0000 7fdde2c0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:27.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.752+0000 7fdde2c0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:27.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.752+0000 7fdde2c0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:07:27.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.752+0000 7fdde2c0f640 1 -- 172.21.15.71:0/1033067971 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdddc1484d0 con 0x7fdddc149dc0 2024-03-29T11:07:27.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.752+0000 7fddd17fa640 1 -- 172.21.15.71:0/1033067971 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7fddcc01a8e0 con 0x7fdddc149dc0 2024-03-29T11:07:27.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.753+0000 7fddd17fa640 1 -- 172.21.15.71:0/1033067971 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7fddcc0544f0 con 0x7fdddc149dc0 2024-03-29T11:07:27.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.753+0000 7fddd17fa640 10 client.4526.objecter ms_dispatch 0x7fdddc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:07:27.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.753+0000 7fddd17fa640 3 client.4526.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:07:27.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.753+0000 7fddd17fa640 3 client.4526.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:07:27.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.753+0000 7fddd17fa640 20 client.4526.objecter dump_active .. 0 homeless 2024-03-29T11:07:27.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.754+0000 7fdde09ab640 10 monclient: get_auth_request con 0x7fddc806dee0 auth_method 0 2024-03-29T11:07:27.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.755+0000 7fddd17fa640 10 client.4526.objecter ms_handle_connect 0x7fddc806dee0 2024-03-29T11:07:27.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.762+0000 7fddd17fa640 1 -- 172.21.15.71:0/1033067971 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fddcc050790 con 0x7fdddc149dc0 2024-03-29T11:07:27.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.762+0000 7fddd17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:27.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:27.762+0000 7fddd17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:07:28.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:28.097+0000 7fdde2c0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:28.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:28.097+0000 7fdde2c0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:28.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:28.097+0000 7fdde2c0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:07:28.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:28.097+0000 7fdde2c0f640 1 -- 172.21.15.71:0/1033067971 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdddc1db510 con 0x7fdddc149dc0 2024-03-29T11:07:28.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:28.099+0000 7fddd17fa640 1 -- 172.21.15.71:0/1033067971 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7fddcc050790 con 0x7fdddc149dc0 2024-03-29T11:07:28.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:28.099+0000 7fddd17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:28.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:28.099+0000 7fddd17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:07:28.078 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:07:28.078 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:07:28.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:28.102+0000 7fdde2c0f640 10 librados: watch_flush enter 2024-03-29T11:07:28.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:28.103+0000 7fdde2c0f640 10 librados: watch_flush exit 2024-03-29T11:07:28.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:28.103+0000 7fdde2c0f640 20 client.4526.objecter shutdown clearing up homeless session... 2024-03-29T11:07:28.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:28.103+0000 7fdde2c0f640 10 client.4526.objecter successfully canceled tick 2024-03-29T11:07:28.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:28.103+0000 7fdde2c0f640 1 -- 172.21.15.71:0/1033067971 >> v1:172.21.15.137:6816/973332665 conn(0x7fddc806dee0 legacy=0x7fddc80703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:28.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:28.103+0000 7fdde2c0f640 10 monclient: shutdown 2024-03-29T11:07:28.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:28.103+0000 7fdde2c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:28.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:28.103+0000 7fdde2c0f640 1 -- 172.21.15.71:0/1033067971 >> v1:172.21.15.71:6790/0 conn(0x7fdddc149dc0 legacy=0x7fdddc1dbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:28.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:28.103+0000 7fdde2c0f640 1 -- 172.21.15.71:0/1033067971 shutdown_connections 2024-03-29T11:07:28.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:28.103+0000 7fdde2c0f640 1 -- 172.21.15.71:0/1033067971 >> 172.21.15.71:0/1033067971 conn(0x7fdddc0b35e0 msgr2=0x7fdddc0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:28.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:28.103+0000 7fdde2c0f640 1 -- 172.21.15.71:0/1033067971 shutdown_connections 2024-03-29T11:07:28.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:28.103+0000 7fdde2c0f640 1 -- 172.21.15.71:0/1033067971 wait complete. 2024-03-29T11:07:28.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:28.103+0000 7fdde2c0f640 1 librados: shutdown 2024-03-29T11:07:28.099 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:07:34.906 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:07:35.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.076+0000 7fa28c8ae640 10 monclient: get_monmap_and_config 2024-03-29T11:07:35.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.076+0000 7fa28c8ae640 10 monclient: build_initial_monmap 2024-03-29T11:07:35.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.076+0000 7fa28c8ae640 10 monclient: monmap: 2024-03-29T11:07:35.056 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:07:35.056 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:35.057 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:07:35.077497+0000 2024-03-29T11:07:35.057 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:07:35.077497+0000 2024-03-29T11:07:35.057 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:35.057 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:35.057 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:07:35.057 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:07:35.057 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:07:35.057 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:35.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.077+0000 7fa28c8ae640 1 Processor -- start 2024-03-29T11:07:35.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.078+0000 7fa28c8ae640 1 -- start start 2024-03-29T11:07:35.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.078+0000 7fa28c8ae640 10 monclient: init 2024-03-29T11:07:35.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.078+0000 7fa28c8ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:35.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.078+0000 7fa28c8ae640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:07:35.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.078+0000 7fa28c8ae640 10 monclient(hunting): picked mon.noname-a con 0x7fa2880b4100 addr v1:172.21.15.71:6789/0 2024-03-29T11:07:35.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.084+0000 7fa28659c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fa2880b4100 0x7fa2880adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:45097/0 (socket says 172.21.15.71:45097) 2024-03-29T11:07:35.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.084+0000 7fa27ffff640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fa28813ee90 0x7fa28813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:46429/0 (socket says 172.21.15.71:46429) 2024-03-29T11:07:35.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.084+0000 7fa28c8ae640 10 monclient(hunting): picked mon.noname-c con 0x7fa28813ee90 addr v1:172.21.15.137:6789/0 2024-03-29T11:07:35.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.084+0000 7fa28c8ae640 10 monclient(hunting): picked mon.noname-b con 0x7fa2880af900 addr v1:172.21.15.71:6790/0 2024-03-29T11:07:35.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.084+0000 7fa28c8ae640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2880afd10 con 0x7fa2880b4100 2024-03-29T11:07:35.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.084+0000 7fa28c8ae640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2880af6f0 con 0x7fa28813ee90 2024-03-29T11:07:35.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.084+0000 7fa28c8ae640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2880aac80 con 0x7fa2880af900 2024-03-29T11:07:35.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.084+0000 7fa28c8ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:35.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.084+0000 7fa28c8ae640 10 monclient(hunting): authenticate will time out at 1211.342568s 2024-03-29T11:07:35.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.084+0000 7fa28659c640 1 -- 172.21.15.71:0/4097547447 learned_addr learned my addr 172.21.15.71:0/4097547447 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:07:35.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.084+0000 7fa285d9b640 1 -- 172.21.15.71:0/4097547447 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2417728732 0 0) 0x7fa2880afd10 con 0x7fa2880b4100 2024-03-29T11:07:35.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.084+0000 7fa285d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:07:35.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.084+0000 7fa285d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:35.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.084+0000 7fa285d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:35.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.084+0000 7fa285d9b640 10 monclient(hunting): my global_id is 4479 2024-03-29T11:07:35.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.084+0000 7fa285d9b640 1 -- 172.21.15.71:0/4097547447 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa274003520 con 0x7fa2880b4100 2024-03-29T11:07:35.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.085+0000 7fa285d9b640 1 -- 172.21.15.71:0/4097547447 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2382108108 0 0) 0x7fa2880af6f0 con 0x7fa28813ee90 2024-03-29T11:07:35.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.085+0000 7fa285d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:07:35.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.085+0000 7fa285d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:35.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.085+0000 7fa285d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:35.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.085+0000 7fa285d9b640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:07:35.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.085+0000 7fa285d9b640 1 -- 172.21.15.71:0/4097547447 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2880afd10 con 0x7fa28813ee90 2024-03-29T11:07:35.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.085+0000 7fa285d9b640 1 -- 172.21.15.71:0/4097547447 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183726596 0 0) 0x7fa2880aac80 con 0x7fa2880af900 2024-03-29T11:07:35.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.085+0000 7fa285d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:07:35.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.085+0000 7fa285d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:35.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.085+0000 7fa285d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:35.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.085+0000 7fa285d9b640 10 monclient(hunting): my global_id is 4529 2024-03-29T11:07:35.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.085+0000 7fa285d9b640 1 -- 172.21.15.71:0/4097547447 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2880af6f0 con 0x7fa2880af900 2024-03-29T11:07:35.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.085+0000 7fa285d9b640 1 -- 172.21.15.71:0/4097547447 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3236499108 0 0) 0x7fa2880af6f0 con 0x7fa2880af900 2024-03-29T11:07:35.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.085+0000 7fa285d9b640 1 -- 172.21.15.71:0/4097547447 >> v1:172.21.15.137:6789/0 conn(0x7fa28813ee90 legacy=0x7fa28813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:35.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.086+0000 7fa285d9b640 1 -- 172.21.15.71:0/4097547447 >> v1:172.21.15.71:6789/0 conn(0x7fa2880b4100 legacy=0x7fa2880adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:35.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.086+0000 7fa285d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:07:35.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.086+0000 7fa285d9b640 1 monclient: found mon.noname-b 2024-03-29T11:07:35.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.086+0000 7fa285d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:35.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.086+0000 7fa285d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:07:35.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.086+0000 7fa285d9b640 1 -- 172.21.15.71:0/4097547447 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2880aaf60 con 0x7fa2880af900 2024-03-29T11:07:35.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.086+0000 7fa285d9b640 10 monclient: _finish_auth 0 2024-03-29T11:07:35.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.086+0000 7fa285d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:07:35.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.086+0000 7fa285d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:35.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.086+0000 7fa285d9b640 1 -- 172.21.15.71:0/4097547447 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fa280001930 con 0x7fa2880af900 2024-03-29T11:07:35.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.086+0000 7fa28c8ae640 5 monclient: authenticate success, global_id 4529 2024-03-29T11:07:35.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.086+0000 7fa28c8ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:07:35.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.086+0000 7fa285d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:35.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.086+0000 7fa285d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:07:35.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.086+0000 7fa285d9b640 10 monclient: dump: 2024-03-29T11:07:35.071 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:35.071 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:35.071 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:35.071 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:35.071 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:35.072 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:35.072 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:35.072 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:35.072 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:35.072 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:35.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.086+0000 7fa28c8ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:07:35.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.087+0000 7fa285d9b640 1 -- 172.21.15.71:0/4097547447 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa280001b50 con 0x7fa2880af900 2024-03-29T11:07:35.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.087+0000 7fa285d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:35.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.087+0000 7fa285d9b640 1 -- 172.21.15.71:0/4097547447 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fa280001ee0 con 0x7fa2880af900 2024-03-29T11:07:35.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.087+0000 7fa28c8ae640 10 monclient: get_monmap_and_config success 2024-03-29T11:07:35.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.087+0000 7fa28c8ae640 4 set_mon_vals no callback set 2024-03-29T11:07:35.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.088+0000 7fa28c8ae640 10 monclient: shutdown 2024-03-29T11:07:35.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.088+0000 7fa285d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:35.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.088+0000 7fa285d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:07:35.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.088+0000 7fa285d9b640 10 monclient: dump: 2024-03-29T11:07:35.073 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:35.073 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:35.073 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:35.073 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:35.073 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:35.073 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:35.073 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:35.073 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:35.073 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:35.073 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:35.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.088+0000 7fa28c8ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:35.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.088+0000 7fa28c8ae640 1 -- 172.21.15.71:0/4097547447 >> v1:172.21.15.71:6790/0 conn(0x7fa2880af900 legacy=0x7fa28814be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:35.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.088+0000 7fa28c8ae640 1 -- 172.21.15.71:0/4097547447 shutdown_connections 2024-03-29T11:07:35.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.088+0000 7fa28c8ae640 1 -- 172.21.15.71:0/4097547447 >> 172.21.15.71:0/4097547447 conn(0x7fa2880b35e0 msgr2=0x7fa2880b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:35.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.088+0000 7fa28c8ae640 1 -- 172.21.15.71:0/4097547447 shutdown_connections 2024-03-29T11:07:35.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.088+0000 7fa28c8ae640 1 -- 172.21.15.71:0/4097547447 wait complete. 2024-03-29T11:07:35.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.090+0000 7fa28c8ae640 10 monclient: build_initial_monmap 2024-03-29T11:07:35.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.090+0000 7fa28c8ae640 10 monclient: monmap: 2024-03-29T11:07:35.075 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:07:35.075 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:07:35.076 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:07:35.076 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:07:35.076 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:35.076 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:35.076 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:07:35.076 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:07:35.076 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:07:35.076 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:35.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.090+0000 7fa28c8ae640 1 librados: starting msgr at 2024-03-29T11:07:35.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.090+0000 7fa28c8ae640 1 librados: starting objecter 2024-03-29T11:07:35.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.090+0000 7fa28c8ae640 1 Processor -- start 2024-03-29T11:07:35.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.090+0000 7fa28c8ae640 1 -- start start 2024-03-29T11:07:35.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.090+0000 7fa28c8ae640 1 librados: setting wanted keys 2024-03-29T11:07:35.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.090+0000 7fa28c8ae640 1 librados: calling monclient init 2024-03-29T11:07:35.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.090+0000 7fa28c8ae640 10 monclient: init 2024-03-29T11:07:35.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.091+0000 7fa28c8ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:35.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.091+0000 7fa28c8ae640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:07:35.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.093+0000 7fa28c8ae640 10 monclient(hunting): picked mon.noname-a con 0x7fa2880af900 addr v1:172.21.15.71:6789/0 2024-03-29T11:07:35.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.094+0000 7fa28c8ae640 10 monclient(hunting): picked mon.noname-b con 0x7fa2880b4100 addr v1:172.21.15.137:6789/0 2024-03-29T11:07:35.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.094+0000 7fa28c8ae640 10 monclient(hunting): picked mon.noname-c con 0x7fa28813ee90 addr v1:172.21.15.71:6790/0 2024-03-29T11:07:35.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.094+0000 7fa28c8ae640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2880726c0 con 0x7fa2880af900 2024-03-29T11:07:35.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.094+0000 7fa28c8ae640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2881d72f0 con 0x7fa2880b4100 2024-03-29T11:07:35.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.094+0000 7fa28c8ae640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa288140a90 con 0x7fa28813ee90 2024-03-29T11:07:35.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.094+0000 7fa28c8ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:35.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.094+0000 7fa28c8ae640 10 monclient(hunting): authenticate will time out at 1211.352895s 2024-03-29T11:07:35.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.094+0000 7fa28659c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fa2880af900 0x7fa288071fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:45345/0 (socket says 172.21.15.71:45345) 2024-03-29T11:07:35.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.094+0000 7fa28659c640 1 -- 172.21.15.71:0/1311325643 learned_addr learned my addr 172.21.15.71:0/1311325643 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:07:35.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.096+0000 7fa27effd640 10 client.?.objecter ms_handle_connect 0x7fa2880af900 2024-03-29T11:07:35.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.096+0000 7fa27effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:35.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.096+0000 7fa27effd640 10 client.?.objecter ms_handle_connect 0x7fa28813ee90 2024-03-29T11:07:35.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.096+0000 7fa27effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:35.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.096+0000 7fa27effd640 1 -- 172.21.15.71:0/1311325643 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839113297 0 0) 0x7fa2880726c0 con 0x7fa2880af900 2024-03-29T11:07:35.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.096+0000 7fa27effd640 10 monclient(hunting): _negotiate 2024-03-29T11:07:35.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.096+0000 7fa27effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:35.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.096+0000 7fa27effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:35.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.096+0000 7fa27effd640 10 monclient(hunting): my global_id is 4482 2024-03-29T11:07:35.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.097+0000 7fa27effd640 1 -- 172.21.15.71:0/1311325643 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa268004eb0 con 0x7fa2880af900 2024-03-29T11:07:35.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.097+0000 7fa27effd640 1 -- 172.21.15.71:0/1311325643 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281908143 0 0) 0x7fa288140a90 con 0x7fa28813ee90 2024-03-29T11:07:35.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.097+0000 7fa27effd640 10 monclient(hunting): _negotiate 2024-03-29T11:07:35.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.097+0000 7fa27effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:35.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.097+0000 7fa27effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:35.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.097+0000 7fa27effd640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:07:35.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.097+0000 7fa27effd640 1 -- 172.21.15.71:0/1311325643 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2880726c0 con 0x7fa28813ee90 2024-03-29T11:07:35.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.097+0000 7fa27effd640 1 -- 172.21.15.71:0/1311325643 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2715210189 0 0) 0x7fa268004eb0 con 0x7fa2880af900 2024-03-29T11:07:35.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.097+0000 7fa27effd640 1 -- 172.21.15.71:0/1311325643 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa288140a90 con 0x7fa2880af900 2024-03-29T11:07:35.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.097+0000 7fa27effd640 1 -- 172.21.15.71:0/1311325643 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3656330353 0 0) 0x7fa2880726c0 con 0x7fa28813ee90 2024-03-29T11:07:35.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.097+0000 7fa27effd640 1 -- 172.21.15.71:0/1311325643 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa268004eb0 con 0x7fa28813ee90 2024-03-29T11:07:35.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.097+0000 7fa27effd640 1 -- 172.21.15.71:0/1311325643 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fa2780013b0 con 0x7fa2880af900 2024-03-29T11:07:35.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.097+0000 7fa27effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:35.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.097+0000 7fa27effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:07:35.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.097+0000 7fa27effd640 10 monclient(hunting): dump: 2024-03-29T11:07:35.083 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:35.083 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:35.083 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:35.083 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:35.083 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:35.083 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:35.083 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:35.083 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:35.083 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:35.083 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:35.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.097+0000 7fa27effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:07:35.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.097+0000 7fa27effd640 1 -- 172.21.15.71:0/1311325643 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fa280001870 con 0x7fa28813ee90 2024-03-29T11:07:35.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.097+0000 7fa27effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:35.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.097+0000 7fa27effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:07:35.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.097+0000 7fa27effd640 10 monclient(hunting): dump: 2024-03-29T11:07:35.084 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:35.084 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:35.084 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:35.084 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:35.084 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:35.084 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:35.084 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:35.084 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:35.084 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:35.084 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:35.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.098+0000 7fa27effd640 1 -- 172.21.15.71:0/1311325643 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 584338939 0 0) 0x7fa288140a90 con 0x7fa2880af900 2024-03-29T11:07:35.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.098+0000 7fa27effd640 1 -- 172.21.15.71:0/1311325643 >> v1:172.21.15.71:6790/0 conn(0x7fa28813ee90 legacy=0x7fa288138750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:35.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.098+0000 7fa27effd640 1 -- 172.21.15.71:0/1311325643 >> v1:172.21.15.137:6789/0 conn(0x7fa2880b4100 legacy=0x7fa2881d6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:35.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.098+0000 7fa27effd640 10 monclient: _finish_hunting 0 2024-03-29T11:07:35.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.098+0000 7fa27effd640 1 monclient: found mon.a 2024-03-29T11:07:35.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.098+0000 7fa27effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:35.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.098+0000 7fa27effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:07:35.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.098+0000 7fa27effd640 1 -- 172.21.15.71:0/1311325643 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2881d66c0 con 0x7fa2880af900 2024-03-29T11:07:35.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.098+0000 7fa27effd640 10 monclient: _finish_auth 0 2024-03-29T11:07:35.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.098+0000 7fa27effd640 10 monclient: _check_auth_tickets 2024-03-29T11:07:35.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.098+0000 7fa27effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:35.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.098+0000 7fa28c8ae640 5 monclient: authenticate success, global_id 4482 2024-03-29T11:07:35.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.098+0000 7fa28c8ae640 10 monclient: _renew_subs 2024-03-29T11:07:35.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.098+0000 7fa27effd640 1 -- 172.21.15.71:0/1311325643 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa278001510 con 0x7fa2880af900 2024-03-29T11:07:35.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.098+0000 7fa28c8ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:07:35.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.098+0000 7fa28c8ae640 1 -- 172.21.15.71:0/1311325643 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa28813f5c0 con 0x7fa2880af900 2024-03-29T11:07:35.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.098+0000 7fa27effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:35.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.098+0000 7fa27effd640 1 -- 172.21.15.71:0/1311325643 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fa278004c30 con 0x7fa2880af900 2024-03-29T11:07:35.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.098+0000 7fa27effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:35.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.098+0000 7fa27effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:07:35.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.098+0000 7fa27effd640 10 monclient: dump: 2024-03-29T11:07:35.086 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:35.086 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:35.086 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:35.086 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:35.086 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:35.086 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:35.086 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:35.086 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:35.086 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:35.087 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:35.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.098+0000 7fa28c8ae640 10 client.4482.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:07:35.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.098+0000 7fa28c8ae640 10 monclient: _renew_subs 2024-03-29T11:07:35.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.098+0000 7fa28c8ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:07:35.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.098+0000 7fa28c8ae640 1 -- 172.21.15.71:0/1311325643 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa28806ef00 con 0x7fa2880af900 2024-03-29T11:07:35.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.098+0000 7fa28c8ae640 1 librados: init done 2024-03-29T11:07:35.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.098+0000 7fa28559a640 4 set_mon_vals no callback set 2024-03-29T11:07:35.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.100+0000 7fa27cff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:35.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.100+0000 7fa27cff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:35.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.100+0000 7fa27cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:07:35.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.100+0000 7fa27cff9640 1 -- 172.21.15.71:0/1311325643 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2881d68b0 con 0x7fa2880af900 2024-03-29T11:07:35.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.100+0000 7fa27effd640 1 -- 172.21.15.71:0/1311325643 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7fa27801ac00 con 0x7fa2880af900 2024-03-29T11:07:35.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.101+0000 7fa27effd640 1 -- 172.21.15.71:0/1311325643 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7fa278084a40 con 0x7fa2880af900 2024-03-29T11:07:35.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.101+0000 7fa27effd640 10 client.4482.objecter ms_dispatch 0x7fa288000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:07:35.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.101+0000 7fa27effd640 3 client.4482.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:07:35.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.101+0000 7fa27effd640 3 client.4482.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:07:35.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.101+0000 7fa27effd640 20 client.4482.objecter dump_active .. 0 homeless 2024-03-29T11:07:35.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.106+0000 7fa27ffff640 10 monclient: get_auth_request con 0x7fa26806e070 auth_method 0 2024-03-29T11:07:35.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.108+0000 7fa27effd640 10 client.4482.objecter ms_handle_connect 0x7fa26806e070 2024-03-29T11:07:35.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.108+0000 7fa27effd640 1 -- 172.21.15.71:0/1311325643 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa278050320 con 0x7fa2880af900 2024-03-29T11:07:35.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.109+0000 7fa27effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:35.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.109+0000 7fa27effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:07:35.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.436+0000 7fa27cff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:35.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.436+0000 7fa27cff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:35.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.437+0000 7fa27cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:07:35.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.437+0000 7fa27cff9640 1 -- 172.21.15.71:0/1311325643 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2881d64b0 con 0x7fa2880af900 2024-03-29T11:07:35.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.439+0000 7fa27effd640 1 -- 172.21.15.71:0/1311325643 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7fa278052250 con 0x7fa2880af900 2024-03-29T11:07:35.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.439+0000 7fa27effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:35.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.439+0000 7fa27effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:07:35.419 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:07:35.419 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:07:35.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.443+0000 7fa27cff9640 10 librados: watch_flush enter 2024-03-29T11:07:35.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.443+0000 7fa27cff9640 10 librados: watch_flush exit 2024-03-29T11:07:35.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.443+0000 7fa27cff9640 20 client.4482.objecter shutdown clearing up homeless session... 2024-03-29T11:07:35.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.443+0000 7fa27cff9640 10 client.4482.objecter successfully canceled tick 2024-03-29T11:07:35.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.443+0000 7fa27cff9640 1 -- 172.21.15.71:0/1311325643 >> v1:172.21.15.137:6816/973332665 conn(0x7fa26806e070 legacy=0x7fa268070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:35.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.443+0000 7fa27cff9640 10 monclient: shutdown 2024-03-29T11:07:35.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.443+0000 7fa27cff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:35.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.443+0000 7fa27cff9640 1 -- 172.21.15.71:0/1311325643 >> v1:172.21.15.71:6789/0 conn(0x7fa2880af900 legacy=0x7fa288071fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:35.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.444+0000 7fa27cff9640 1 -- 172.21.15.71:0/1311325643 shutdown_connections 2024-03-29T11:07:35.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.444+0000 7fa27cff9640 1 -- 172.21.15.71:0/1311325643 >> 172.21.15.71:0/1311325643 conn(0x7fa2880b35e0 msgr2=0x7fa2880acf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:35.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.444+0000 7fa27cff9640 1 -- 172.21.15.71:0/1311325643 shutdown_connections 2024-03-29T11:07:35.425 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.446+0000 7fa27cff9640 1 -- 172.21.15.71:0/1311325643 wait complete. 2024-03-29T11:07:35.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:35.448+0000 7fa27cff9640 1 librados: shutdown 2024-03-29T11:07:35.447 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:07:42.255 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:07:42.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.423+0000 7f308af3b640 10 monclient: get_monmap_and_config 2024-03-29T11:07:42.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.423+0000 7f308af3b640 10 monclient: build_initial_monmap 2024-03-29T11:07:42.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.423+0000 7f308af3b640 10 monclient: monmap: 2024-03-29T11:07:42.402 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:07:42.403 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:42.403 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:07:42.424481+0000 2024-03-29T11:07:42.403 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:07:42.424481+0000 2024-03-29T11:07:42.403 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:42.403 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:42.403 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:07:42.403 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:07:42.403 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:07:42.403 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:42.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.426+0000 7f308af3b640 1 Processor -- start 2024-03-29T11:07:42.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.426+0000 7f308af3b640 1 -- start start 2024-03-29T11:07:42.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.426+0000 7f308af3b640 10 monclient: init 2024-03-29T11:07:42.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.427+0000 7f308af3b640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:42.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.427+0000 7f308af3b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:07:42.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.427+0000 7f308af3b640 10 monclient(hunting): picked mon.noname-b con 0x7f307c006890 addr v1:172.21.15.71:6790/0 2024-03-29T11:07:42.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.427+0000 7f308af3b640 10 monclient(hunting): picked mon.noname-c con 0x7f307c006c70 addr v1:172.21.15.137:6789/0 2024-03-29T11:07:42.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.427+0000 7f308af3b640 10 monclient(hunting): picked mon.noname-a con 0x7f307c02b4c0 addr v1:172.21.15.71:6789/0 2024-03-29T11:07:42.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.427+0000 7f308af3b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f307c02c830 con 0x7f307c02b4c0 2024-03-29T11:07:42.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.427+0000 7f308af3b640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f307c02afa0 con 0x7f307c006c70 2024-03-29T11:07:42.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.427+0000 7f308af3b640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f307c02aba0 con 0x7f307c006890 2024-03-29T11:07:42.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.428+0000 7f308af3b640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:42.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.428+0000 7f308af3b640 10 monclient(hunting): authenticate will time out at 1218.686120s 2024-03-29T11:07:42.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.432+0000 7f3089f39640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f307c006890 0x7f307c0c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:50540/0 (socket says 172.21.15.71:50540) 2024-03-29T11:07:42.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.432+0000 7f3089f39640 1 -- 172.21.15.71:0/1426578978 learned_addr learned my addr 172.21.15.71:0/1426578978 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:07:42.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.433+0000 7f3088f37640 1 -- 172.21.15.71:0/1426578978 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1650034820 0 0) 0x7f307c02c830 con 0x7f307c02b4c0 2024-03-29T11:07:42.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.433+0000 7f3088f37640 10 monclient(hunting): _negotiate 2024-03-29T11:07:42.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.433+0000 7f3088f37640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:42.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.433+0000 7f3088f37640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:42.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.433+0000 7f3088f37640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:07:42.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.433+0000 7f3088f37640 1 -- 172.21.15.71:0/1426578978 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3070003520 con 0x7f307c02b4c0 2024-03-29T11:07:42.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.434+0000 7f3088f37640 1 -- 172.21.15.71:0/1426578978 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 581208088 0 0) 0x7f3070003520 con 0x7f307c02b4c0 2024-03-29T11:07:42.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.434+0000 7f3088f37640 1 -- 172.21.15.71:0/1426578978 >> v1:172.21.15.71:6790/0 conn(0x7f307c006890 legacy=0x7f307c0c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:07:42.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.434+0000 7f3088f37640 1 -- 172.21.15.71:0/1426578978 >> v1:172.21.15.137:6789/0 conn(0x7f307c006c70 legacy=0x7f307c0c6d40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:07:42.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.434+0000 7f3088f37640 10 monclient: _finish_hunting 0 2024-03-29T11:07:42.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.434+0000 7f3088f37640 1 monclient: found mon.noname-a 2024-03-29T11:07:42.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.434+0000 7f3088f37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:42.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.434+0000 7f3088f37640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:07:42.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.434+0000 7f3088f37640 1 -- 172.21.15.71:0/1426578978 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307c02ad30 con 0x7f307c02b4c0 2024-03-29T11:07:42.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.434+0000 7f3088f37640 10 monclient: _finish_auth 0 2024-03-29T11:07:42.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.434+0000 7f3088f37640 10 monclient: _check_auth_tickets 2024-03-29T11:07:42.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.434+0000 7f3088f37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:42.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.434+0000 7f3088f37640 1 -- 172.21.15.71:0/1426578978 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f308404f730 con 0x7f307c02b4c0 2024-03-29T11:07:42.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.434+0000 7f3088f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:42.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.434+0000 7f3088f37640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:07:42.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.434+0000 7f3088f37640 10 monclient: dump: 2024-03-29T11:07:42.416 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:42.416 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:42.416 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:42.416 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:42.416 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:42.417 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:42.417 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:42.417 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:42.417 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:42.417 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:42.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.434+0000 7f3088f37640 1 -- 172.21.15.71:0/1426578978 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30840008d0 con 0x7f307c02b4c0 2024-03-29T11:07:42.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.434+0000 7f3088f37640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:42.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.434+0000 7f3088f37640 1 -- 172.21.15.71:0/1426578978 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f30840758b0 con 0x7f307c02b4c0 2024-03-29T11:07:42.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.434+0000 7f3088f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:42.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.434+0000 7f3088f37640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:07:42.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.434+0000 7f3088f37640 10 monclient: dump: 2024-03-29T11:07:42.417 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:42.417 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:42.417 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:42.418 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:42.418 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:42.418 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:42.418 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:42.418 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:42.418 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:42.418 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:42.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.434+0000 7f308af3b640 5 monclient: authenticate success, global_id 4485 2024-03-29T11:07:42.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.434+0000 7f308af3b640 10 monclient: get_monmap_and_config success 2024-03-29T11:07:42.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.434+0000 7f308af3b640 4 set_mon_vals no callback set 2024-03-29T11:07:42.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.436+0000 7f308af3b640 10 monclient: shutdown 2024-03-29T11:07:42.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.436+0000 7f308af3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:42.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.436+0000 7f308af3b640 1 -- 172.21.15.71:0/1426578978 >> v1:172.21.15.71:6789/0 conn(0x7f307c02b4c0 legacy=0x7f307c0cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:42.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.436+0000 7f308af3b640 1 -- 172.21.15.71:0/1426578978 shutdown_connections 2024-03-29T11:07:42.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.436+0000 7f308af3b640 1 -- 172.21.15.71:0/1426578978 >> 172.21.15.71:0/1426578978 conn(0x7f307c0058f0 msgr2=0x7f307c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:42.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.436+0000 7f308af3b640 1 -- 172.21.15.71:0/1426578978 shutdown_connections 2024-03-29T11:07:42.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.437+0000 7f308af3b640 1 -- 172.21.15.71:0/1426578978 wait complete. 2024-03-29T11:07:42.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.437+0000 7f308af3b640 10 monclient: build_initial_monmap 2024-03-29T11:07:42.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.437+0000 7f308af3b640 10 monclient: monmap: 2024-03-29T11:07:42.420 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:07:42.420 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:07:42.420 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:07:42.420 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:07:42.420 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:42.420 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:42.420 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:07:42.420 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:07:42.420 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:07:42.421 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:42.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.438+0000 7f308af3b640 1 librados: starting msgr at 2024-03-29T11:07:42.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.438+0000 7f308af3b640 1 librados: starting objecter 2024-03-29T11:07:42.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.438+0000 7f308af3b640 1 Processor -- start 2024-03-29T11:07:42.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.438+0000 7f308af3b640 1 -- start start 2024-03-29T11:07:42.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.438+0000 7f308af3b640 1 librados: setting wanted keys 2024-03-29T11:07:42.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.438+0000 7f308af3b640 1 librados: calling monclient init 2024-03-29T11:07:42.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.438+0000 7f308af3b640 10 monclient: init 2024-03-29T11:07:42.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.439+0000 7f308af3b640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:42.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.439+0000 7f308af3b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:07:42.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.439+0000 7f308af3b640 10 monclient(hunting): picked mon.noname-b con 0x7f307c006890 addr v1:172.21.15.137:6789/0 2024-03-29T11:07:42.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.440+0000 7f308af3b640 10 monclient(hunting): picked mon.noname-a con 0x7f307c02b4c0 addr v1:172.21.15.71:6789/0 2024-03-29T11:07:42.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.440+0000 7f308af3b640 10 monclient(hunting): picked mon.noname-c con 0x7f307c14f9e0 addr v1:172.21.15.71:6790/0 2024-03-29T11:07:42.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.440+0000 7f308af3b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f307c164d20 con 0x7f307c02b4c0 2024-03-29T11:07:42.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.440+0000 7f308af3b640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f307c164a60 con 0x7f307c006890 2024-03-29T11:07:42.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.440+0000 7f308af3b640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f307c0c42b0 con 0x7f307c14f9e0 2024-03-29T11:07:42.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.440+0000 7f308af3b640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:42.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.440+0000 7f308af3b640 10 monclient(hunting): authenticate will time out at 1218.698204s 2024-03-29T11:07:42.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.441+0000 7f308a73a640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f307c14f9e0 0x7f307c0c1ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:51936/0 (socket says 172.21.15.71:51936) 2024-03-29T11:07:42.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.441+0000 7f308a73a640 1 -- 172.21.15.71:0/4239715518 learned_addr learned my addr 172.21.15.71:0/4239715518 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:07:42.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.442+0000 7f307a7fc640 10 client.?.objecter ms_handle_connect 0x7f307c14f9e0 2024-03-29T11:07:42.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.442+0000 7f307a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:42.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.442+0000 7f307a7fc640 1 -- 172.21.15.71:0/4239715518 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3591362689 0 0) 0x7f307c0c42b0 con 0x7f307c14f9e0 2024-03-29T11:07:42.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.442+0000 7f307a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:07:42.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.442+0000 7f307a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:42.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.442+0000 7f307a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:42.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.442+0000 7f307a7fc640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:07:42.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.442+0000 7f307a7fc640 1 -- 172.21.15.71:0/4239715518 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3068004eb0 con 0x7f307c14f9e0 2024-03-29T11:07:42.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.442+0000 7f307a7fc640 1 -- 172.21.15.71:0/4239715518 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1201135668 0 0) 0x7f3068004eb0 con 0x7f307c14f9e0 2024-03-29T11:07:42.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.442+0000 7f307a7fc640 1 -- 172.21.15.71:0/4239715518 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f307c0c42b0 con 0x7f307c14f9e0 2024-03-29T11:07:42.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.442+0000 7f307a7fc640 1 -- 172.21.15.71:0/4239715518 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f308404f400 con 0x7f307c14f9e0 2024-03-29T11:07:42.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.442+0000 7f307a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:42.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.442+0000 7f307a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:07:42.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.442+0000 7f307a7fc640 10 monclient(hunting): dump: 2024-03-29T11:07:42.424 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:42.424 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:42.424 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:42.424 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:42.424 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:42.424 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:42.424 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:42.424 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:42.424 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:42.424 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:42.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.442+0000 7f307a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:07:42.425 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.442+0000 7f307a7fc640 1 -- 172.21.15.71:0/4239715518 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4032882615 0 0) 0x7f307c0c42b0 con 0x7f307c14f9e0 2024-03-29T11:07:42.425 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.442+0000 7f307a7fc640 1 -- 172.21.15.71:0/4239715518 >> v1:172.21.15.137:6789/0 conn(0x7f307c006890 legacy=0x7f307c164350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:07:42.425 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.443+0000 7f307a7fc640 1 -- 172.21.15.71:0/4239715518 >> v1:172.21.15.71:6789/0 conn(0x7f307c02b4c0 legacy=0x7f307c14f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:42.425 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.443+0000 7f307a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:07:42.425 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.443+0000 7f307a7fc640 1 monclient: found mon.c 2024-03-29T11:07:42.425 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.443+0000 7f307a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:42.425 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.443+0000 7f307a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:07:42.425 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.443+0000 7f307a7fc640 1 -- 172.21.15.71:0/4239715518 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307c0c4500 con 0x7f307c14f9e0 2024-03-29T11:07:42.425 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.443+0000 7f307a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:07:42.425 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.443+0000 7f307a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:07:42.425 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.443+0000 7f307a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:42.425 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.444+0000 7f307a7fc640 1 -- 172.21.15.71:0/4239715518 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f308404f730 con 0x7f307c14f9e0 2024-03-29T11:07:42.425 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.444+0000 7f307a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:42.425 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.444+0000 7f307a7fc640 1 -- 172.21.15.71:0/4239715518 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f308404f400 con 0x7f307c14f9e0 2024-03-29T11:07:42.425 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.444+0000 7f307a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:42.426 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.444+0000 7f307a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:07:42.426 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.444+0000 7f307a7fc640 10 monclient: dump: 2024-03-29T11:07:42.426 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:42.426 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:42.426 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:42.426 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:42.426 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:42.426 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:42.426 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:42.426 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:42.426 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:42.426 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:42.426 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.444+0000 7f307bfff640 4 set_mon_vals no callback set 2024-03-29T11:07:42.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.445+0000 7f308af3b640 5 monclient: authenticate success, global_id 4535 2024-03-29T11:07:42.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.445+0000 7f308af3b640 10 monclient: _renew_subs 2024-03-29T11:07:42.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.445+0000 7f308af3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:07:42.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.445+0000 7f308af3b640 1 -- 172.21.15.71:0/4239715518 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f307c14ff50 con 0x7f307c14f9e0 2024-03-29T11:07:42.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.445+0000 7f308af3b640 10 client.4535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:07:42.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.445+0000 7f308af3b640 10 monclient: _renew_subs 2024-03-29T11:07:42.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.445+0000 7f308af3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:07:42.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.445+0000 7f308af3b640 1 -- 172.21.15.71:0/4239715518 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f307c0c25b0 con 0x7f307c14f9e0 2024-03-29T11:07:42.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.445+0000 7f308af3b640 1 librados: init done 2024-03-29T11:07:42.428 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.448+0000 7f308af3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:42.428 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.448+0000 7f308af3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:42.428 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.448+0000 7f308af3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:07:42.428 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.448+0000 7f308af3b640 1 -- 172.21.15.71:0/4239715518 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f307c004cd0 con 0x7f307c14f9e0 2024-03-29T11:07:42.430 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.450+0000 7f307a7fc640 1 -- 172.21.15.71:0/4239715518 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f3084067bf0 con 0x7f307c14f9e0 2024-03-29T11:07:42.430 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.451+0000 7f307a7fc640 1 -- 172.21.15.71:0/4239715518 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f30840ff430 con 0x7f307c14f9e0 2024-03-29T11:07:42.430 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.451+0000 7f307a7fc640 10 client.4535.objecter ms_dispatch 0x7f3084000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:07:42.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.451+0000 7f307a7fc640 3 client.4535.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:07:42.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.451+0000 7f307a7fc640 3 client.4535.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:07:42.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.451+0000 7f307a7fc640 20 client.4535.objecter dump_active .. 0 homeless 2024-03-29T11:07:42.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.452+0000 7f3089f39640 10 monclient: get_auth_request con 0x7f306806dcc0 auth_method 0 2024-03-29T11:07:42.431 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.452+0000 7f307a7fc640 10 client.4535.objecter ms_handle_connect 0x7f306806dcc0 2024-03-29T11:07:42.438 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.459+0000 7f307a7fc640 1 -- 172.21.15.71:0/4239715518 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f30840cad70 con 0x7f307c14f9e0 2024-03-29T11:07:42.438 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.460+0000 7f307a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:42.438 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.460+0000 7f307a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:07:42.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.790+0000 7f308af3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:42.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.790+0000 7f308af3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:42.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.790+0000 7f308af3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:07:42.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.790+0000 7f308af3b640 1 -- 172.21.15.71:0/4239715518 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f307c164ed0 con 0x7f307c14f9e0 2024-03-29T11:07:42.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.792+0000 7f307a7fc640 1 -- 172.21.15.71:0/4239715518 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f307c164ed0 con 0x7f307c14f9e0 2024-03-29T11:07:42.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.792+0000 7f307a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:42.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.792+0000 7f307a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:07:42.943 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:07:42.943 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:07:42.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.796+0000 7f308af3b640 10 librados: watch_flush enter 2024-03-29T11:07:42.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.796+0000 7f308af3b640 10 librados: watch_flush exit 2024-03-29T11:07:42.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.796+0000 7f308af3b640 20 client.4535.objecter shutdown clearing up homeless session... 2024-03-29T11:07:42.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.796+0000 7f308af3b640 10 client.4535.objecter successfully canceled tick 2024-03-29T11:07:42.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.796+0000 7f308af3b640 1 -- 172.21.15.71:0/4239715518 >> v1:172.21.15.137:6816/973332665 conn(0x7f306806dcc0 legacy=0x7f3068070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:42.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.796+0000 7f308af3b640 10 monclient: shutdown 2024-03-29T11:07:42.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.796+0000 7f308af3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:42.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.796+0000 7f308af3b640 1 -- 172.21.15.71:0/4239715518 >> v1:172.21.15.71:6790/0 conn(0x7f307c14f9e0 legacy=0x7f307c0c1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:42.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.796+0000 7f308af3b640 1 -- 172.21.15.71:0/4239715518 shutdown_connections 2024-03-29T11:07:42.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.796+0000 7f308af3b640 1 -- 172.21.15.71:0/4239715518 >> 172.21.15.71:0/4239715518 conn(0x7f307c0058f0 msgr2=0x7f307c038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:42.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.796+0000 7f308af3b640 1 -- 172.21.15.71:0/4239715518 shutdown_connections 2024-03-29T11:07:42.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.796+0000 7f308af3b640 1 -- 172.21.15.71:0/4239715518 wait complete. 2024-03-29T11:07:42.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:42.797+0000 7f308af3b640 1 librados: shutdown 2024-03-29T11:07:42.946 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:07:49.759 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:07:49.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.944+0000 7f9e268cf640 10 monclient: get_monmap_and_config 2024-03-29T11:07:49.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.944+0000 7f9e268cf640 10 monclient: build_initial_monmap 2024-03-29T11:07:49.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.944+0000 7f9e268cf640 10 monclient: monmap: 2024-03-29T11:07:49.925 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:07:49.925 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:49.925 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:07:49.946653+0000 2024-03-29T11:07:49.925 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:07:49.946653+0000 2024-03-29T11:07:49.925 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:49.925 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:49.925 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:07:49.925 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:07:49.925 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:07:49.926 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:49.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.946+0000 7f9e268cf640 1 Processor -- start 2024-03-29T11:07:49.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.946+0000 7f9e268cf640 1 -- start start 2024-03-29T11:07:49.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.946+0000 7f9e268cf640 10 monclient: init 2024-03-29T11:07:49.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.947+0000 7f9e268cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:49.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.947+0000 7f9e268cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:07:49.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.947+0000 7f9e268cf640 10 monclient(hunting): picked mon.noname-b con 0x7f9e200b0500 addr v1:172.21.15.71:6790/0 2024-03-29T11:07:49.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.947+0000 7f9e268cf640 10 monclient(hunting): picked mon.noname-a con 0x7f9e200b08e0 addr v1:172.21.15.71:6789/0 2024-03-29T11:07:49.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.951+0000 7f9e268cf640 10 monclient(hunting): picked mon.noname-c con 0x7f9e200aeca0 addr v1:172.21.15.137:6789/0 2024-03-29T11:07:49.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.953+0000 7f9e268cf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e20140620 con 0x7f9e200b08e0 2024-03-29T11:07:49.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.953+0000 7f9e268cf640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e2013daf0 con 0x7f9e200aeca0 2024-03-29T11:07:49.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.953+0000 7f9e268cf640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e200ae180 con 0x7f9e200b0500 2024-03-29T11:07:49.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.953+0000 7f9e268cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:49.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.953+0000 7f9e268cf640 10 monclient(hunting): authenticate will time out at 1226.212388s 2024-03-29T11:07:49.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.953+0000 7f9e17fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f9e200b08e0 0x7f9e2013d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:44671/0 (socket says 172.21.15.71:44671) 2024-03-29T11:07:49.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.953+0000 7f9e17fff640 1 -- 172.21.15.71:0/1535631266 learned_addr learned my addr 172.21.15.71:0/1535631266 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:07:49.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.958+0000 7f9e1f7fe640 1 -- 172.21.15.71:0/1535631266 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3270085120 0 0) 0x7f9e20140620 con 0x7f9e200b08e0 2024-03-29T11:07:49.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.958+0000 7f9e1f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:07:49.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.958+0000 7f9e1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:49.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.958+0000 7f9e1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:49.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.958+0000 7f9e1f7fe640 10 monclient(hunting): my global_id is 4488 2024-03-29T11:07:49.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.958+0000 7f9e1f7fe640 1 -- 172.21.15.71:0/1535631266 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e0c003520 con 0x7f9e200b08e0 2024-03-29T11:07:49.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.959+0000 7f9e1f7fe640 1 -- 172.21.15.71:0/1535631266 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156824785 0 0) 0x7f9e2013daf0 con 0x7f9e200aeca0 2024-03-29T11:07:49.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.959+0000 7f9e1f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:07:49.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.959+0000 7f9e1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:49.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.959+0000 7f9e1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:49.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.959+0000 7f9e1f7fe640 10 monclient(hunting): my global_id is 4444 2024-03-29T11:07:49.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.959+0000 7f9e1f7fe640 1 -- 172.21.15.71:0/1535631266 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e0c003840 con 0x7f9e200aeca0 2024-03-29T11:07:49.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.960+0000 7f9e1f7fe640 1 -- 172.21.15.71:0/1535631266 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2300627444 0 0) 0x7f9e200ae180 con 0x7f9e200b0500 2024-03-29T11:07:49.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.960+0000 7f9e1f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:07:49.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.960+0000 7f9e1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:49.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.960+0000 7f9e1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:49.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.960+0000 7f9e1f7fe640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:07:49.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.960+0000 7f9e1f7fe640 1 -- 172.21.15.71:0/1535631266 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e2013daf0 con 0x7f9e200b0500 2024-03-29T11:07:49.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.961+0000 7f9e1f7fe640 1 -- 172.21.15.71:0/1535631266 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2657618679 0 0) 0x7f9e0c003840 con 0x7f9e200aeca0 2024-03-29T11:07:49.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.961+0000 7f9e1f7fe640 1 -- 172.21.15.71:0/1535631266 >> v1:172.21.15.71:6790/0 conn(0x7f9e200b0500 legacy=0x7f9e2013ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:49.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.961+0000 7f9e1f7fe640 1 -- 172.21.15.71:0/1535631266 >> v1:172.21.15.71:6789/0 conn(0x7f9e200b08e0 legacy=0x7f9e2013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:49.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.961+0000 7f9e1f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:07:49.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.961+0000 7f9e1f7fe640 1 monclient: found mon.noname-c 2024-03-29T11:07:49.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.961+0000 7f9e1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:49.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.961+0000 7f9e1f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.137:6789/0 2024-03-29T11:07:49.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.961+0000 7f9e1f7fe640 1 -- 172.21.15.71:0/1535631266 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e200accf0 con 0x7f9e200aeca0 2024-03-29T11:07:49.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.961+0000 7f9e1f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:07:49.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.961+0000 7f9e1f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:07:49.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.961+0000 7f9e1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:49.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.961+0000 7f9e1f7fe640 1 -- 172.21.15.71:0/1535631266 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9e180019f0 con 0x7f9e200aeca0 2024-03-29T11:07:49.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.961+0000 7f9e1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:49.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.961+0000 7f9e1f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:07:49.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.961+0000 7f9e1f7fe640 10 monclient: dump: 2024-03-29T11:07:49.943 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:49.943 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:49.943 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:49.943 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:49.943 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:49.943 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:49.943 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:49.943 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:49.943 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:49.943 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:49.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.962+0000 7f9e268cf640 5 monclient: authenticate success, global_id 4444 2024-03-29T11:07:49.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.962+0000 7f9e268cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:07:49.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.962+0000 7f9e1f7fe640 1 -- 172.21.15.71:0/1535631266 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e18001b50 con 0x7f9e200aeca0 2024-03-29T11:07:49.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.962+0000 7f9e1f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:49.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.962+0000 7f9e268cf640 10 monclient: get_monmap_and_config success 2024-03-29T11:07:49.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.962+0000 7f9e268cf640 4 set_mon_vals no callback set 2024-03-29T11:07:49.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.963+0000 7f9e268cf640 10 monclient: shutdown 2024-03-29T11:07:49.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.963+0000 7f9e1f7fe640 1 -- 172.21.15.71:0/1535631266 <== mon.1 v1:172.21.15.137:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9e18001fa0 con 0x7f9e200aeca0 2024-03-29T11:07:49.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.963+0000 7f9e1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:49.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.963+0000 7f9e1f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:07:49.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.963+0000 7f9e1f7fe640 10 monclient: dump: 2024-03-29T11:07:49.945 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:49.945 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:49.945 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:49.945 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:49.945 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:49.945 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:49.945 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:49.945 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:49.945 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:49.945 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:49.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.963+0000 7f9e268cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:49.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.963+0000 7f9e268cf640 1 -- 172.21.15.71:0/1535631266 >> v1:172.21.15.137:6789/0 conn(0x7f9e200aeca0 legacy=0x7f9e2014b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:49.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.965+0000 7f9e268cf640 1 -- 172.21.15.71:0/1535631266 shutdown_connections 2024-03-29T11:07:49.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.965+0000 7f9e268cf640 1 -- 172.21.15.71:0/1535631266 >> 172.21.15.71:0/1535631266 conn(0x7f9e2006b660 msgr2=0x7f9e2006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:49.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.965+0000 7f9e268cf640 1 -- 172.21.15.71:0/1535631266 shutdown_connections 2024-03-29T11:07:49.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.968+0000 7f9e268cf640 1 -- 172.21.15.71:0/1535631266 wait complete. 2024-03-29T11:07:49.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.969+0000 7f9e268cf640 10 monclient: build_initial_monmap 2024-03-29T11:07:49.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.969+0000 7f9e268cf640 10 monclient: monmap: 2024-03-29T11:07:49.949 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:07:49.949 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:07:49.949 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:07:49.949 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:07:49.949 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:49.949 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:49.949 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:07:49.950 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:07:49.950 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:07:49.950 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:49.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.969+0000 7f9e268cf640 1 librados: starting msgr at 2024-03-29T11:07:49.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.969+0000 7f9e268cf640 1 librados: starting objecter 2024-03-29T11:07:49.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.969+0000 7f9e268cf640 1 Processor -- start 2024-03-29T11:07:49.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.970+0000 7f9e268cf640 1 -- start start 2024-03-29T11:07:49.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.970+0000 7f9e268cf640 1 librados: setting wanted keys 2024-03-29T11:07:49.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.970+0000 7f9e268cf640 1 librados: calling monclient init 2024-03-29T11:07:49.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.970+0000 7f9e268cf640 10 monclient: init 2024-03-29T11:07:49.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.971+0000 7f9e268cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:49.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.971+0000 7f9e268cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:07:49.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.973+0000 7f9e268cf640 10 monclient(hunting): picked mon.noname-a con 0x7f9e200aeca0 addr v1:172.21.15.71:6789/0 2024-03-29T11:07:49.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.973+0000 7f9e268cf640 10 monclient(hunting): picked mon.noname-c con 0x7f9e200b0500 addr v1:172.21.15.71:6790/0 2024-03-29T11:07:49.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.973+0000 7f9e268cf640 10 monclient(hunting): picked mon.noname-b con 0x7f9e2012eca0 addr v1:172.21.15.137:6789/0 2024-03-29T11:07:49.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.973+0000 7f9e268cf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e201e3fc0 con 0x7f9e200aeca0 2024-03-29T11:07:49.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.973+0000 7f9e268cf640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e2012cfc0 con 0x7f9e2012eca0 2024-03-29T11:07:49.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.973+0000 7f9e268cf640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e201e4b60 con 0x7f9e200b0500 2024-03-29T11:07:49.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.973+0000 7f9e268cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:49.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.973+0000 7f9e268cf640 10 monclient(hunting): authenticate will time out at 1226.232865s 2024-03-29T11:07:49.957 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.977+0000 7f9e24e6c640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f9e2012eca0 0x7f9e201ef850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:43873/0 (socket says 172.21.15.71:43873) 2024-03-29T11:07:49.957 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.977+0000 7f9e24e6c640 1 -- 172.21.15.71:0/2163559990 learned_addr learned my addr 172.21.15.71:0/2163559990 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:07:49.965 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.981+0000 7f9e1d7fa640 10 client.?.objecter ms_handle_connect 0x7f9e2012eca0 2024-03-29T11:07:49.965 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.981+0000 7f9e1d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:49.965 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.981+0000 7f9e1d7fa640 10 client.?.objecter ms_handle_connect 0x7f9e200aeca0 2024-03-29T11:07:49.965 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.981+0000 7f9e1d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:49.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.982+0000 7f9e1d7fa640 1 -- 172.21.15.71:0/2163559990 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431861190 0 0) 0x7f9e201e3fc0 con 0x7f9e200aeca0 2024-03-29T11:07:49.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.982+0000 7f9e1d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:07:49.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.982+0000 7f9e1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:49.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.982+0000 7f9e1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:49.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.982+0000 7f9e1d7fa640 10 monclient(hunting): my global_id is 4491 2024-03-29T11:07:49.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.982+0000 7f9e1d7fa640 1 -- 172.21.15.71:0/2163559990 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e00004eb0 con 0x7f9e200aeca0 2024-03-29T11:07:49.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.982+0000 7f9e1d7fa640 1 -- 172.21.15.71:0/2163559990 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2794964795 0 0) 0x7f9e2012cfc0 con 0x7f9e2012eca0 2024-03-29T11:07:49.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.982+0000 7f9e1d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:07:49.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.982+0000 7f9e1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:49.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.982+0000 7f9e1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:49.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.982+0000 7f9e1d7fa640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:07:49.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.982+0000 7f9e1d7fa640 1 -- 172.21.15.71:0/2163559990 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e201e3fc0 con 0x7f9e2012eca0 2024-03-29T11:07:49.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.982+0000 7f9e1d7fa640 10 client.?.objecter ms_handle_connect 0x7f9e200b0500 2024-03-29T11:07:49.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.982+0000 7f9e1d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:49.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.982+0000 7f9e1d7fa640 1 -- 172.21.15.71:0/2163559990 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 186648414 0 0) 0x7f9e00004eb0 con 0x7f9e200aeca0 2024-03-29T11:07:49.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.982+0000 7f9e1d7fa640 1 -- 172.21.15.71:0/2163559990 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e2012cfc0 con 0x7f9e200aeca0 2024-03-29T11:07:49.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.982+0000 7f9e1d7fa640 1 -- 172.21.15.71:0/2163559990 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9e10001880 con 0x7f9e200aeca0 2024-03-29T11:07:49.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.982+0000 7f9e1d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:49.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.982+0000 7f9e1d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:07:49.967 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.982+0000 7f9e1d7fa640 10 monclient(hunting): dump: 2024-03-29T11:07:49.967 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:49.967 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:49.967 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:49.967 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:49.967 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:49.967 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:49.968 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:49.968 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:49.968 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:49.968 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:49.968 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.983+0000 7f9e1d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:07:49.968 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.983+0000 7f9e1d7fa640 1 -- 172.21.15.71:0/2163559990 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2997877993 0 0) 0x7f9e201e3fc0 con 0x7f9e2012eca0 2024-03-29T11:07:49.968 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.983+0000 7f9e1d7fa640 1 -- 172.21.15.71:0/2163559990 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e00004eb0 con 0x7f9e2012eca0 2024-03-29T11:07:49.968 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.983+0000 7f9e1d7fa640 1 -- 172.21.15.71:0/2163559990 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9e18001ea0 con 0x7f9e2012eca0 2024-03-29T11:07:49.968 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.983+0000 7f9e1d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:49.968 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.983+0000 7f9e1d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:07:49.968 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.983+0000 7f9e1d7fa640 10 monclient(hunting): dump: 2024-03-29T11:07:49.968 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:49.968 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:49.969 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:49.969 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:49.969 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:49.969 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:49.969 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:49.969 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:49.969 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:49.969 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:49.969 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.983+0000 7f9e1d7fa640 1 -- 172.21.15.71:0/2163559990 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157568615 0 0) 0x7f9e201e4b60 con 0x7f9e200b0500 2024-03-29T11:07:49.969 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.983+0000 7f9e1d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:07:49.969 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.983+0000 7f9e1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:49.969 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.983+0000 7f9e1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:49.969 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.983+0000 7f9e1d7fa640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:07:49.970 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.983+0000 7f9e1d7fa640 1 -- 172.21.15.71:0/2163559990 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e201e3fc0 con 0x7f9e200b0500 2024-03-29T11:07:49.970 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.983+0000 7f9e1d7fa640 1 -- 172.21.15.71:0/2163559990 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 997512569 0 0) 0x7f9e00004eb0 con 0x7f9e2012eca0 2024-03-29T11:07:49.970 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.983+0000 7f9e1d7fa640 1 -- 172.21.15.71:0/2163559990 >> v1:172.21.15.71:6790/0 conn(0x7f9e200b0500 legacy=0x7f9e201e1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:49.970 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.983+0000 7f9e1d7fa640 1 -- 172.21.15.71:0/2163559990 >> v1:172.21.15.71:6789/0 conn(0x7f9e200aeca0 legacy=0x7f9e201e4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:49.970 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.984+0000 7f9e1d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:07:49.970 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.984+0000 7f9e1d7fa640 1 monclient: found mon.b 2024-03-29T11:07:49.970 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.984+0000 7f9e1d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:49.970 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.984+0000 7f9e1d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:07:49.970 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.984+0000 7f9e1d7fa640 1 -- 172.21.15.71:0/2163559990 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e201e4cf0 con 0x7f9e2012eca0 2024-03-29T11:07:49.970 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.984+0000 7f9e1d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:07:49.970 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.984+0000 7f9e1d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:07:49.970 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.984+0000 7f9e1d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:49.970 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.984+0000 7f9e268cf640 5 monclient: authenticate success, global_id 4447 2024-03-29T11:07:49.970 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.984+0000 7f9e268cf640 10 monclient: _renew_subs 2024-03-29T11:07:49.971 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.984+0000 7f9e268cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:07:49.971 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.984+0000 7f9e268cf640 1 -- 172.21.15.71:0/2163559990 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e2012fa30 con 0x7f9e2012eca0 2024-03-29T11:07:49.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.984+0000 7f9e268cf640 10 client.4447.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:07:49.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.984+0000 7f9e268cf640 10 monclient: _renew_subs 2024-03-29T11:07:49.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.984+0000 7f9e268cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:07:49.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.984+0000 7f9e268cf640 1 -- 172.21.15.71:0/2163559990 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e201e1750 con 0x7f9e2012eca0 2024-03-29T11:07:49.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.984+0000 7f9e268cf640 1 librados: init done 2024-03-29T11:07:49.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.986+0000 7f9e167fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:49.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.986+0000 7f9e167fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:49.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.986+0000 7f9e167fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:07:49.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.986+0000 7f9e167fc640 1 -- 172.21.15.71:0/2163559990 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e200ad0c0 con 0x7f9e2012eca0 2024-03-29T11:07:49.973 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.988+0000 7f9e1d7fa640 1 -- 172.21.15.71:0/2163559990 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e18003c20 con 0x7f9e2012eca0 2024-03-29T11:07:49.973 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.988+0000 7f9e1d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:49.973 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.988+0000 7f9e1d7fa640 1 -- 172.21.15.71:0/2163559990 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f9e18004030 con 0x7f9e2012eca0 2024-03-29T11:07:49.973 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.988+0000 7f9e1d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:49.973 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.988+0000 7f9e1d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:07:49.973 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.988+0000 7f9e1d7fa640 10 monclient: dump: 2024-03-29T11:07:49.973 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:49.973 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:49.973 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:49.973 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:49.973 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:49.973 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:49.973 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:49.973 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:49.973 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:49.974 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:49.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.988+0000 7f9e1e7fc640 4 set_mon_vals no callback set 2024-03-29T11:07:49.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.990+0000 7f9e1d7fa640 1 -- 172.21.15.71:0/2163559990 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f9e18004190 con 0x7f9e2012eca0 2024-03-29T11:07:49.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.991+0000 7f9e1d7fa640 1 -- 172.21.15.71:0/2163559990 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f9e18084dd0 con 0x7f9e2012eca0 2024-03-29T11:07:49.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.991+0000 7f9e1d7fa640 10 client.4447.objecter ms_dispatch 0x7f9e20000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:07:49.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.991+0000 7f9e1d7fa640 3 client.4447.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:07:49.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.991+0000 7f9e1d7fa640 3 client.4447.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:07:49.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.991+0000 7f9e1d7fa640 20 client.4447.objecter dump_active .. 0 homeless 2024-03-29T11:07:49.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.992+0000 7f9e1ffff640 10 monclient: get_auth_request con 0x7f9e0006dff0 auth_method 0 2024-03-29T11:07:49.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.992+0000 7f9e1d7fa640 10 client.4447.objecter ms_handle_connect 0x7f9e0006dff0 2024-03-29T11:07:49.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.998+0000 7f9e1d7fa640 1 -- 172.21.15.71:0/2163559990 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9e180506b0 con 0x7f9e2012eca0 2024-03-29T11:07:49.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.998+0000 7f9e1d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:49.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:49.998+0000 7f9e1d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:07:50.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:50.331+0000 7f9e167fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:50.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:50.331+0000 7f9e167fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:50.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:50.331+0000 7f9e167fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:07:50.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:50.331+0000 7f9e167fc640 1 -- 172.21.15.71:0/2163559990 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e20075f10 con 0x7f9e2012eca0 2024-03-29T11:07:50.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:50.334+0000 7f9e1d7fa640 1 -- 172.21.15.71:0/2163559990 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f9e180525e0 con 0x7f9e2012eca0 2024-03-29T11:07:50.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:50.334+0000 7f9e1d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:50.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:50.334+0000 7f9e1d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:07:50.316 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:07:50.316 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:07:50.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:50.338+0000 7f9e167fc640 10 librados: watch_flush enter 2024-03-29T11:07:50.318 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:50.338+0000 7f9e167fc640 10 librados: watch_flush exit 2024-03-29T11:07:50.318 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:50.338+0000 7f9e167fc640 20 client.4447.objecter shutdown clearing up homeless session... 2024-03-29T11:07:50.318 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:50.338+0000 7f9e167fc640 10 client.4447.objecter successfully canceled tick 2024-03-29T11:07:50.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:50.344+0000 7f9e167fc640 1 -- 172.21.15.71:0/2163559990 >> v1:172.21.15.137:6816/973332665 conn(0x7f9e0006dff0 legacy=0x7f9e000704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:50.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:50.344+0000 7f9e167fc640 10 monclient: shutdown 2024-03-29T11:07:50.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:50.344+0000 7f9e167fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:50.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:50.344+0000 7f9e167fc640 1 -- 172.21.15.71:0/2163559990 >> v1:172.21.15.137:6789/0 conn(0x7f9e2012eca0 legacy=0x7f9e201ef850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:50.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:50.344+0000 7f9e167fc640 1 -- 172.21.15.71:0/2163559990 shutdown_connections 2024-03-29T11:07:50.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:50.344+0000 7f9e167fc640 1 -- 172.21.15.71:0/2163559990 >> 172.21.15.71:0/2163559990 conn(0x7f9e2006b660 msgr2=0x7f9e2006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:50.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:50.344+0000 7f9e167fc640 1 -- 172.21.15.71:0/2163559990 shutdown_connections 2024-03-29T11:07:50.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:50.345+0000 7f9e167fc640 1 -- 172.21.15.71:0/2163559990 wait complete. 2024-03-29T11:07:50.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:50.345+0000 7f9e167fc640 1 librados: shutdown 2024-03-29T11:07:50.341 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:07:57.148 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:07:57.297 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.318+0000 7f3ea31a5640 10 monclient: get_monmap_and_config 2024-03-29T11:07:57.297 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.318+0000 7f3ea31a5640 10 monclient: build_initial_monmap 2024-03-29T11:07:57.298 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.319+0000 7f3ea31a5640 10 monclient: monmap: 2024-03-29T11:07:57.298 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:07:57.298 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:57.298 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:07:57.319893+0000 2024-03-29T11:07:57.298 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:07:57.319893+0000 2024-03-29T11:07:57.298 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:57.298 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:57.298 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:07:57.298 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:07:57.298 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:07:57.298 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:57.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.320+0000 7f3ea31a5640 1 Processor -- start 2024-03-29T11:07:57.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.320+0000 7f3ea31a5640 1 -- start start 2024-03-29T11:07:57.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.320+0000 7f3ea31a5640 10 monclient: init 2024-03-29T11:07:57.301 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.321+0000 7f3ea31a5640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:57.301 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.321+0000 7f3ea31a5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:07:57.301 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.321+0000 7f3ea31a5640 10 monclient(hunting): picked mon.noname-a con 0x7f3e9c070170 addr v1:172.21.15.71:6789/0 2024-03-29T11:07:57.301 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.321+0000 7f3ea31a5640 10 monclient(hunting): picked mon.noname-c con 0x7f3e9c0713f0 addr v1:172.21.15.137:6789/0 2024-03-29T11:07:57.301 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.321+0000 7f3ea31a5640 10 monclient(hunting): picked mon.noname-b con 0x7f3e9c070550 addr v1:172.21.15.71:6790/0 2024-03-29T11:07:57.301 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.321+0000 7f3ea31a5640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e9c13dbd0 con 0x7f3e9c070170 2024-03-29T11:07:57.301 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.321+0000 7f3ea31a5640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e9c13ff70 con 0x7f3e9c0713f0 2024-03-29T11:07:57.301 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.321+0000 7f3ea31a5640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e9c140250 con 0x7f3e9c070550 2024-03-29T11:07:57.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.321+0000 7f3ea31a5640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:57.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.321+0000 7f3ea31a5640 10 monclient(hunting): authenticate will time out at 1233.579299s 2024-03-29T11:07:57.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.325+0000 7f3e9bfff640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f3e9c0713f0 0x7f3e9c1406e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:54099/0 (socket says 172.21.15.71:54099) 2024-03-29T11:07:57.304 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.325+0000 7f3e9bfff640 1 -- 172.21.15.71:0/1186709071 learned_addr learned my addr 172.21.15.71:0/1186709071 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:07:57.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 1 -- 172.21.15.71:0/1186709071 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662844377 0 0) 0x7f3e90001110 con 0x7f3e9c070170 2024-03-29T11:07:57.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:07:57.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:57.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:57.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:07:57.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 1 -- 172.21.15.71:0/1186709071 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e84003520 con 0x7f3e9c070170 2024-03-29T11:07:57.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 1 -- 172.21.15.71:0/1186709071 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3647080489 0 0) 0x7f3e9c140250 con 0x7f3e9c070550 2024-03-29T11:07:57.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:07:57.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:57.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:57.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:07:57.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 1 -- 172.21.15.71:0/1186709071 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e90001110 con 0x7f3e9c070550 2024-03-29T11:07:57.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 1 -- 172.21.15.71:0/1186709071 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3712332963 0 0) 0x7f3e84003520 con 0x7f3e9c070170 2024-03-29T11:07:57.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 1 -- 172.21.15.71:0/1186709071 >> v1:172.21.15.71:6790/0 conn(0x7f3e9c070550 legacy=0x7f3e9c14b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:57.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 1 -- 172.21.15.71:0/1186709071 >> v1:172.21.15.137:6789/0 conn(0x7f3e9c0713f0 legacy=0x7f3e9c1406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:57.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:07:57.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 1 monclient: found mon.noname-a 2024-03-29T11:07:57.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:57.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:07:57.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 1 -- 172.21.15.71:0/1186709071 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e9c140df0 con 0x7f3e9c070170 2024-03-29T11:07:57.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:07:57.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:07:57.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:57.310 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3ea31a5640 5 monclient: authenticate success, global_id 4494 2024-03-29T11:07:57.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3ea31a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:07:57.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 1 -- 172.21.15.71:0/1186709071 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f3e90001680 con 0x7f3e9c070170 2024-03-29T11:07:57.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:57.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:07:57.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 10 monclient: dump: 2024-03-29T11:07:57.311 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:57.311 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:57.311 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:57.311 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:57.311 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:57.311 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:57.311 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:57.311 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:57.311 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:57.311 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:57.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3ea31a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:07:57.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 1 -- 172.21.15.71:0/1186709071 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e900018a0 con 0x7f3e9c070170 2024-03-29T11:07:57.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3e9b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:57.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3ea31a5640 10 monclient: get_monmap_and_config success 2024-03-29T11:07:57.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.326+0000 7f3ea31a5640 4 set_mon_vals no callback set 2024-03-29T11:07:57.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.327+0000 7f3ea31a5640 10 monclient: shutdown 2024-03-29T11:07:57.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.327+0000 7f3ea31a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:57.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.327+0000 7f3ea31a5640 1 -- 172.21.15.71:0/1186709071 >> v1:172.21.15.71:6789/0 conn(0x7f3e9c070170 legacy=0x7f3e9c13d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:57.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.328+0000 7f3ea31a5640 1 -- 172.21.15.71:0/1186709071 shutdown_connections 2024-03-29T11:07:57.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.328+0000 7f3ea31a5640 1 -- 172.21.15.71:0/1186709071 >> 172.21.15.71:0/1186709071 conn(0x7f3e9c067610 msgr2=0x7f3e9c0b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:57.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.328+0000 7f3ea31a5640 1 -- 172.21.15.71:0/1186709071 shutdown_connections 2024-03-29T11:07:57.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.328+0000 7f3ea31a5640 1 -- 172.21.15.71:0/1186709071 wait complete. 2024-03-29T11:07:57.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.329+0000 7f3ea31a5640 10 monclient: build_initial_monmap 2024-03-29T11:07:57.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.329+0000 7f3ea31a5640 10 monclient: monmap: 2024-03-29T11:07:57.313 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:07:57.313 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:07:57.313 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:07:57.313 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:07:57.313 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:07:57.313 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:57.313 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:07:57.313 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:07:57.313 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:07:57.313 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:57.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.330+0000 7f3ea31a5640 1 librados: starting msgr at 2024-03-29T11:07:57.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.330+0000 7f3ea31a5640 1 librados: starting objecter 2024-03-29T11:07:57.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.330+0000 7f3ea31a5640 1 Processor -- start 2024-03-29T11:07:57.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.330+0000 7f3ea31a5640 1 -- start start 2024-03-29T11:07:57.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.330+0000 7f3ea31a5640 1 librados: setting wanted keys 2024-03-29T11:07:57.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.330+0000 7f3ea31a5640 1 librados: calling monclient init 2024-03-29T11:07:57.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.330+0000 7f3ea31a5640 10 monclient: init 2024-03-29T11:07:57.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.331+0000 7f3ea31a5640 10 monclient: _reopen_session rank -1 2024-03-29T11:07:57.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.331+0000 7f3ea31a5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:07:57.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.331+0000 7f3ea31a5640 10 monclient(hunting): picked mon.noname-a con 0x7f3e9c070170 addr v1:172.21.15.71:6789/0 2024-03-29T11:07:57.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.333+0000 7f3ea31a5640 10 monclient(hunting): picked mon.noname-c con 0x7f3e9c070550 addr v1:172.21.15.71:6790/0 2024-03-29T11:07:57.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.333+0000 7f3ea31a5640 10 monclient(hunting): picked mon.noname-b con 0x7f3e9c0713f0 addr v1:172.21.15.137:6789/0 2024-03-29T11:07:57.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.333+0000 7f3ea31a5640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e9c136820 con 0x7f3e9c070170 2024-03-29T11:07:57.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.333+0000 7f3ea31a5640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e9c1369b0 con 0x7f3e9c0713f0 2024-03-29T11:07:57.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.333+0000 7f3ea31a5640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e9c0b4c80 con 0x7f3e9c070550 2024-03-29T11:07:57.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.333+0000 7f3ea31a5640 10 monclient(hunting): _renew_subs 2024-03-29T11:07:57.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.333+0000 7f3ea31a5640 10 monclient(hunting): authenticate will time out at 1233.591633s 2024-03-29T11:07:57.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.333+0000 7f3ea0f41640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f3e9c070170 0x7f3e9c1d6d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:48660/0 (socket says 172.21.15.71:48660) 2024-03-29T11:07:57.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.333+0000 7f3ea0f41640 1 -- 172.21.15.71:0/1407646134 learned_addr learned my addr 172.21.15.71:0/1407646134 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:07:57.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.334+0000 7f3e997fa640 10 client.?.objecter ms_handle_connect 0x7f3e9c070170 2024-03-29T11:07:57.315 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.334+0000 7f3e997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:57.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.335+0000 7f3e997fa640 10 client.?.objecter ms_handle_connect 0x7f3e9c070550 2024-03-29T11:07:57.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.335+0000 7f3e997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:57.321 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.335+0000 7f3e997fa640 1 -- 172.21.15.71:0/1407646134 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180820196 0 0) 0x7f3e9c0b4c80 con 0x7f3e9c070550 2024-03-29T11:07:57.321 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.335+0000 7f3e997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:07:57.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.335+0000 7f3e997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:57.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.335+0000 7f3e997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:57.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.335+0000 7f3e997fa640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:07:57.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.335+0000 7f3e997fa640 1 -- 172.21.15.71:0/1407646134 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e88004eb0 con 0x7f3e9c070550 2024-03-29T11:07:57.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.338+0000 7f3e997fa640 10 client.?.objecter ms_handle_connect 0x7f3e9c0713f0 2024-03-29T11:07:57.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.338+0000 7f3e997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:07:57.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.338+0000 7f3e997fa640 1 -- 172.21.15.71:0/1407646134 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2642085863 0 0) 0x7f3e9c136820 con 0x7f3e9c070170 2024-03-29T11:07:57.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.338+0000 7f3e997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:07:57.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.338+0000 7f3e997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:57.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.338+0000 7f3e997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:57.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.338+0000 7f3e997fa640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:07:57.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.338+0000 7f3e997fa640 1 -- 172.21.15.71:0/1407646134 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e9c0b4c80 con 0x7f3e9c070170 2024-03-29T11:07:57.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.338+0000 7f3e997fa640 1 -- 172.21.15.71:0/1407646134 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1552735635 0 0) 0x7f3e88004eb0 con 0x7f3e9c070550 2024-03-29T11:07:57.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.338+0000 7f3e997fa640 1 -- 172.21.15.71:0/1407646134 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e9c136820 con 0x7f3e9c070550 2024-03-29T11:07:57.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.338+0000 7f3e997fa640 1 -- 172.21.15.71:0/1407646134 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f3e94001ac0 con 0x7f3e9c070550 2024-03-29T11:07:57.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.338+0000 7f3e997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:57.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.338+0000 7f3e997fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:07:57.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.338+0000 7f3e997fa640 10 monclient(hunting): dump: 2024-03-29T11:07:57.323 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:57.323 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:57.323 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:57.323 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:57.323 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:57.323 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:57.323 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:57.323 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:57.323 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:57.323 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:57.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.338+0000 7f3e997fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:07:57.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.338+0000 7f3e997fa640 1 -- 172.21.15.71:0/1407646134 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306883686 0 0) 0x7f3e9c1369b0 con 0x7f3e9c0713f0 2024-03-29T11:07:57.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.338+0000 7f3e997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:07:57.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.338+0000 7f3e997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:07:57.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.338+0000 7f3e997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:07:57.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.338+0000 7f3e997fa640 10 monclient(hunting): my global_id is 4453 2024-03-29T11:07:57.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.338+0000 7f3e997fa640 1 -- 172.21.15.71:0/1407646134 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e88004eb0 con 0x7f3e9c0713f0 2024-03-29T11:07:57.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.338+0000 7f3e997fa640 1 -- 172.21.15.71:0/1407646134 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1387043716 0 0) 0x7f3e88004eb0 con 0x7f3e9c0713f0 2024-03-29T11:07:57.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.338+0000 7f3e997fa640 1 -- 172.21.15.71:0/1407646134 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e9c1369b0 con 0x7f3e9c0713f0 2024-03-29T11:07:57.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.338+0000 7f3e997fa640 1 -- 172.21.15.71:0/1407646134 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f3e8c001930 con 0x7f3e9c0713f0 2024-03-29T11:07:57.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.338+0000 7f3e997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:57.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.338+0000 7f3e997fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:07:57.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.338+0000 7f3e997fa640 10 monclient(hunting): dump: 2024-03-29T11:07:57.324 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:57.325 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:57.325 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:57.325 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:57.325 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:57.325 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:57.325 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:57.325 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:57.325 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:57.325 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:57.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.339+0000 7f3e997fa640 1 -- 172.21.15.71:0/1407646134 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 221677376 0 0) 0x7f3e9c1369b0 con 0x7f3e9c0713f0 2024-03-29T11:07:57.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.339+0000 7f3e997fa640 1 -- 172.21.15.71:0/1407646134 >> v1:172.21.15.71:6790/0 conn(0x7f3e9c070550 legacy=0x7f3e9c1d3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:57.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.339+0000 7f3e997fa640 1 -- 172.21.15.71:0/1407646134 >> v1:172.21.15.71:6789/0 conn(0x7f3e9c070170 legacy=0x7f3e9c1d6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:57.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.339+0000 7f3e997fa640 10 monclient: _finish_hunting 0 2024-03-29T11:07:57.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.339+0000 7f3e997fa640 1 monclient: found mon.b 2024-03-29T11:07:57.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.339+0000 7f3e997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:07:57.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.339+0000 7f3e997fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:07:57.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.339+0000 7f3e997fa640 1 -- 172.21.15.71:0/1407646134 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e9c0b4e10 con 0x7f3e9c0713f0 2024-03-29T11:07:57.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.339+0000 7f3e997fa640 10 monclient: _finish_auth 0 2024-03-29T11:07:57.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.339+0000 7f3e997fa640 10 monclient: _check_auth_tickets 2024-03-29T11:07:57.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.339+0000 7f3e997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:07:57.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.339+0000 7f3ea31a5640 5 monclient: authenticate success, global_id 4453 2024-03-29T11:07:57.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.339+0000 7f3ea31a5640 10 monclient: _renew_subs 2024-03-29T11:07:57.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.339+0000 7f3ea31a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:07:57.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.339+0000 7f3ea31a5640 1 -- 172.21.15.71:0/1407646134 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e9c0b5020 con 0x7f3e9c0713f0 2024-03-29T11:07:57.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.339+0000 7f3ea31a5640 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:07:57.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.339+0000 7f3ea31a5640 10 monclient: _renew_subs 2024-03-29T11:07:57.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.339+0000 7f3ea31a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:07:57.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.339+0000 7f3ea31a5640 1 -- 172.21.15.71:0/1407646134 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e9c1defb0 con 0x7f3e9c0713f0 2024-03-29T11:07:57.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.339+0000 7f3ea31a5640 1 librados: init done 2024-03-29T11:07:57.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.339+0000 7f3e997fa640 1 -- 172.21.15.71:0/1407646134 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e8c001d40 con 0x7f3e9c0713f0 2024-03-29T11:07:57.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.339+0000 7f3e997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:07:57.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.339+0000 7f3e997fa640 1 -- 172.21.15.71:0/1407646134 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f3e8c0037b0 con 0x7f3e9c0713f0 2024-03-29T11:07:57.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.339+0000 7f3e997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:07:57.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.339+0000 7f3e997fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:07:57.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.339+0000 7f3e997fa640 10 monclient: dump: 2024-03-29T11:07:57.327 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:07:57.327 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:07:57.327 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:57.327 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:07:57.327 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:07:57.327 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:07:57.327 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:07:57.327 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:07:57.328 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:07:57.328 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:07:57.328 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.339+0000 7f3e9a7fc640 4 set_mon_vals no callback set 2024-03-29T11:07:57.328 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.346+0000 7f3e997fa640 1 -- 172.21.15.71:0/1407646134 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f3e8c019780 con 0x7f3e9c0713f0 2024-03-29T11:07:57.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.348+0000 7f3e997fa640 1 -- 172.21.15.71:0/1407646134 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f3e8c083750 con 0x7f3e9c0713f0 2024-03-29T11:07:57.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.348+0000 7f3e997fa640 10 client.4453.objecter ms_dispatch 0x7f3e9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:07:57.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.348+0000 7f3ea31a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:57.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.348+0000 7f3e997fa640 3 client.4453.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:07:57.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.348+0000 7f3e997fa640 3 client.4453.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:07:57.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.348+0000 7f3ea31a5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:57.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.348+0000 7f3ea31a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:07:57.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.348+0000 7f3ea31a5640 1 -- 172.21.15.71:0/1407646134 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e7c005080 con 0x7f3e9c0713f0 2024-03-29T11:07:57.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.348+0000 7f3e997fa640 20 client.4453.objecter dump_active .. 0 homeless 2024-03-29T11:07:57.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.348+0000 7f3ea0f41640 10 monclient: get_auth_request con 0x7f3e8806df00 auth_method 0 2024-03-29T11:07:57.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.349+0000 7f3e997fa640 10 client.4453.objecter ms_handle_connect 0x7f3e8806df00 2024-03-29T11:07:57.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.355+0000 7f3e997fa640 1 -- 172.21.15.71:0/1407646134 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3e8c04f090 con 0x7f3e9c0713f0 2024-03-29T11:07:57.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.355+0000 7f3e997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:07:57.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.355+0000 7f3e997fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:07:57.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.741+0000 7f3ea31a5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:57.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.741+0000 7f3ea31a5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:57.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.741+0000 7f3ea31a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:07:57.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.741+0000 7f3ea31a5640 1 -- 172.21.15.71:0/1407646134 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e7c005330 con 0x7f3e9c0713f0 2024-03-29T11:07:57.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.742+0000 7f3e997fa640 1 -- 172.21.15.71:0/1407646134 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f3e8c050fc0 con 0x7f3e9c0713f0 2024-03-29T11:07:57.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.742+0000 7f3e997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:07:57.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.742+0000 7f3e997fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:07:57.722 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:07:57.722 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:07:57.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.748+0000 7f3ea31a5640 10 librados: watch_flush enter 2024-03-29T11:07:57.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.748+0000 7f3ea31a5640 10 librados: watch_flush exit 2024-03-29T11:07:57.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.748+0000 7f3ea31a5640 20 client.4453.objecter shutdown clearing up homeless session... 2024-03-29T11:07:57.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.748+0000 7f3ea31a5640 10 client.4453.objecter successfully canceled tick 2024-03-29T11:07:57.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.749+0000 7f3ea31a5640 1 -- 172.21.15.71:0/1407646134 >> v1:172.21.15.137:6816/973332665 conn(0x7f3e8806df00 legacy=0x7f3e880703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:57.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.749+0000 7f3ea31a5640 10 monclient: shutdown 2024-03-29T11:07:57.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.749+0000 7f3ea31a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:07:57.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.749+0000 7f3ea31a5640 1 -- 172.21.15.71:0/1407646134 >> v1:172.21.15.137:6789/0 conn(0x7f3e9c0713f0 legacy=0x7f3e9c1d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:07:57.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.750+0000 7f3ea31a5640 1 -- 172.21.15.71:0/1407646134 shutdown_connections 2024-03-29T11:07:57.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.750+0000 7f3ea31a5640 1 -- 172.21.15.71:0/1407646134 >> 172.21.15.71:0/1407646134 conn(0x7f3e9c067610 msgr2=0x7f3e9c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:07:57.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.751+0000 7f3ea31a5640 1 -- 172.21.15.71:0/1407646134 shutdown_connections 2024-03-29T11:07:57.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.752+0000 7f3ea31a5640 1 -- 172.21.15.71:0/1407646134 wait complete. 2024-03-29T11:07:57.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:07:57.755+0000 7f3ea31a5640 1 librados: shutdown 2024-03-29T11:07:57.754 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:08:04.560 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:08:04.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.733+0000 7fbe62a4a640 10 monclient: get_monmap_and_config 2024-03-29T11:08:04.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.733+0000 7fbe62a4a640 10 monclient: build_initial_monmap 2024-03-29T11:08:04.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.733+0000 7fbe62a4a640 10 monclient: monmap: 2024-03-29T11:08:04.712 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:08:04.712 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:04.712 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:08:04.734618+0000 2024-03-29T11:08:04.713 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:08:04.734618+0000 2024-03-29T11:08:04.713 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:04.713 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:04.713 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:08:04.713 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:08:04.713 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:08:04.713 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:04.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.734+0000 7fbe62a4a640 1 Processor -- start 2024-03-29T11:08:04.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.734+0000 7fbe62a4a640 1 -- start start 2024-03-29T11:08:04.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.734+0000 7fbe62a4a640 10 monclient: init 2024-03-29T11:08:04.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.735+0000 7fbe62a4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:04.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.736+0000 7fbe62a4a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:08:04.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.736+0000 7fbe62a4a640 10 monclient(hunting): picked mon.noname-c con 0x7fbe5c0b1f60 addr v1:172.21.15.137:6789/0 2024-03-29T11:08:04.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.736+0000 7fbe62a4a640 10 monclient(hunting): picked mon.noname-b con 0x7fbe5c071eb0 addr v1:172.21.15.71:6790/0 2024-03-29T11:08:04.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.736+0000 7fbe62a4a640 10 monclient(hunting): picked mon.noname-a con 0x7fbe5c143d10 addr v1:172.21.15.71:6789/0 2024-03-29T11:08:04.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.736+0000 7fbe62a4a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5c06ef80 con 0x7fbe5c143d10 2024-03-29T11:08:04.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.736+0000 7fbe62a4a640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5c06f260 con 0x7fbe5c0b1f60 2024-03-29T11:08:04.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.736+0000 7fbe62a4a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5c0b40c0 con 0x7fbe5c071eb0 2024-03-29T11:08:04.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.736+0000 7fbe62a4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:04.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.736+0000 7fbe62a4a640 10 monclient(hunting): authenticate will time out at 1240.994216s 2024-03-29T11:08:04.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.738+0000 7fbe5b7fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fbe5c071eb0 0x7fbe5c146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:55103/0 (socket says 172.21.15.71:55103) 2024-03-29T11:08:04.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.738+0000 7fbe5b7fe640 1 -- 172.21.15.71:0/2994011083 learned_addr learned my addr 172.21.15.71:0/2994011083 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:08:04.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.739+0000 7fbe5affd640 1 -- 172.21.15.71:0/2994011083 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955569501 0 0) 0x7fbe5c0b40c0 con 0x7fbe5c071eb0 2024-03-29T11:08:04.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.739+0000 7fbe5affd640 10 monclient(hunting): _negotiate 2024-03-29T11:08:04.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.739+0000 7fbe5affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:04.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.739+0000 7fbe5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:04.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.739+0000 7fbe5affd640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:08:04.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.739+0000 7fbe5affd640 1 -- 172.21.15.71:0/2994011083 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe44003520 con 0x7fbe5c071eb0 2024-03-29T11:08:04.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.740+0000 7fbe5affd640 1 -- 172.21.15.71:0/2994011083 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1201856990 0 0) 0x7fbe44003520 con 0x7fbe5c071eb0 2024-03-29T11:08:04.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.740+0000 7fbe5affd640 1 -- 172.21.15.71:0/2994011083 >> v1:172.21.15.137:6789/0 conn(0x7fbe5c0b1f60 legacy=0x7fbe5c143600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:08:04.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.742+0000 7fbe5affd640 1 -- 172.21.15.71:0/2994011083 >> v1:172.21.15.71:6789/0 conn(0x7fbe5c143d10 legacy=0x7fbe5c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:04.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.742+0000 7fbe5affd640 10 monclient: _finish_hunting 0 2024-03-29T11:08:04.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.742+0000 7fbe5affd640 1 monclient: found mon.noname-b 2024-03-29T11:08:04.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.742+0000 7fbe5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:04.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.742+0000 7fbe5affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:08:04.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.742+0000 7fbe5affd640 1 -- 172.21.15.71:0/2994011083 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe5c155d70 con 0x7fbe5c071eb0 2024-03-29T11:08:04.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.742+0000 7fbe5affd640 10 monclient: _finish_auth 0 2024-03-29T11:08:04.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.742+0000 7fbe5affd640 10 monclient: _check_auth_tickets 2024-03-29T11:08:04.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.742+0000 7fbe5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:04.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.742+0000 7fbe5affd640 1 -- 172.21.15.71:0/2994011083 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fbe540019f0 con 0x7fbe5c071eb0 2024-03-29T11:08:04.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.742+0000 7fbe5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:04.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.742+0000 7fbe5affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:08:04.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.742+0000 7fbe5affd640 10 monclient: dump: 2024-03-29T11:08:04.722 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:04.722 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:04.722 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:04.722 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:04.722 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:04.722 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:04.722 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:04.723 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:04.723 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:04.723 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:04.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.742+0000 7fbe62a4a640 5 monclient: authenticate success, global_id 4550 2024-03-29T11:08:04.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.742+0000 7fbe62a4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:04.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.742+0000 7fbe5affd640 1 -- 172.21.15.71:0/2994011083 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe54001c80 con 0x7fbe5c071eb0 2024-03-29T11:08:04.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.742+0000 7fbe5affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:04.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.742+0000 7fbe62a4a640 10 monclient: get_monmap_and_config success 2024-03-29T11:08:04.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.742+0000 7fbe62a4a640 4 set_mon_vals no callback set 2024-03-29T11:08:04.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.743+0000 7fbe5affd640 1 -- 172.21.15.71:0/2994011083 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fbe54002050 con 0x7fbe5c071eb0 2024-03-29T11:08:04.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.743+0000 7fbe5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:04.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.743+0000 7fbe5affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:04.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.743+0000 7fbe5affd640 10 monclient: dump: 2024-03-29T11:08:04.723 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:04.724 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:04.724 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:04.724 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:04.724 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:04.724 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:04.724 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:04.724 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:04.724 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:04.724 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:04.726 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.743+0000 7fbe62a4a640 10 monclient: shutdown 2024-03-29T11:08:04.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.743+0000 7fbe62a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:04.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.743+0000 7fbe62a4a640 1 -- 172.21.15.71:0/2994011083 >> v1:172.21.15.71:6790/0 conn(0x7fbe5c071eb0 legacy=0x7fbe5c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:04.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.743+0000 7fbe62a4a640 1 -- 172.21.15.71:0/2994011083 shutdown_connections 2024-03-29T11:08:04.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.743+0000 7fbe62a4a640 1 -- 172.21.15.71:0/2994011083 >> 172.21.15.71:0/2994011083 conn(0x7fbe5c066a00 msgr2=0x7fbe5c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:04.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.743+0000 7fbe62a4a640 1 -- 172.21.15.71:0/2994011083 shutdown_connections 2024-03-29T11:08:04.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.743+0000 7fbe62a4a640 1 -- 172.21.15.71:0/2994011083 wait complete. 2024-03-29T11:08:04.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.744+0000 7fbe62a4a640 10 monclient: build_initial_monmap 2024-03-29T11:08:04.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.744+0000 7fbe62a4a640 10 monclient: monmap: 2024-03-29T11:08:04.727 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:08:04.727 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:08:04.727 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:08:04.727 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:08:04.727 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:04.727 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:04.728 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:08:04.728 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:08:04.728 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:08:04.728 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:04.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.745+0000 7fbe62a4a640 1 librados: starting msgr at 2024-03-29T11:08:04.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.745+0000 7fbe62a4a640 1 librados: starting objecter 2024-03-29T11:08:04.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.745+0000 7fbe62a4a640 1 Processor -- start 2024-03-29T11:08:04.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.745+0000 7fbe62a4a640 1 -- start start 2024-03-29T11:08:04.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.745+0000 7fbe62a4a640 1 librados: setting wanted keys 2024-03-29T11:08:04.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.745+0000 7fbe62a4a640 1 librados: calling monclient init 2024-03-29T11:08:04.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.745+0000 7fbe62a4a640 10 monclient: init 2024-03-29T11:08:04.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.746+0000 7fbe62a4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:04.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.746+0000 7fbe62a4a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:08:04.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.746+0000 7fbe62a4a640 10 monclient(hunting): picked mon.noname-c con 0x7fbe5c0b1f60 addr v1:172.21.15.71:6790/0 2024-03-29T11:08:04.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.746+0000 7fbe62a4a640 10 monclient(hunting): picked mon.noname-b con 0x7fbe5c143d10 addr v1:172.21.15.137:6789/0 2024-03-29T11:08:04.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.746+0000 7fbe62a4a640 10 monclient(hunting): picked mon.noname-a con 0x7fbe5c142be0 addr v1:172.21.15.71:6789/0 2024-03-29T11:08:04.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.746+0000 7fbe62a4a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5c137f60 con 0x7fbe5c142be0 2024-03-29T11:08:04.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.746+0000 7fbe62a4a640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5c132740 con 0x7fbe5c143d10 2024-03-29T11:08:04.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.746+0000 7fbe62a4a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5c1541d0 con 0x7fbe5c0b1f60 2024-03-29T11:08:04.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.746+0000 7fbe62a4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:04.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.746+0000 7fbe62a4a640 10 monclient(hunting): authenticate will time out at 1241.004544s 2024-03-29T11:08:04.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.749+0000 7fbe60fe7640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fbe5c142be0 0x7fbe5c132030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:50522/0 (socket says 172.21.15.71:50522) 2024-03-29T11:08:04.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.749+0000 7fbe60fe7640 1 -- 172.21.15.71:0/3680909018 learned_addr learned my addr 172.21.15.71:0/3680909018 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:08:04.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.749+0000 7fbe58ff9640 10 client.?.objecter ms_handle_connect 0x7fbe5c142be0 2024-03-29T11:08:04.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.749+0000 7fbe58ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:04.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.749+0000 7fbe58ff9640 1 -- 172.21.15.71:0/3680909018 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3396724795 0 0) 0x7fbe5c137f60 con 0x7fbe5c142be0 2024-03-29T11:08:04.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.749+0000 7fbe58ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:08:04.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.749+0000 7fbe58ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:04.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.749+0000 7fbe58ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:04.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.749+0000 7fbe58ff9640 10 monclient(hunting): my global_id is 4503 2024-03-29T11:08:04.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.749+0000 7fbe58ff9640 1 -- 172.21.15.71:0/3680909018 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe48004eb0 con 0x7fbe5c142be0 2024-03-29T11:08:04.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.750+0000 7fbe58ff9640 1 -- 172.21.15.71:0/3680909018 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3406473945 0 0) 0x7fbe48004eb0 con 0x7fbe5c142be0 2024-03-29T11:08:04.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.750+0000 7fbe58ff9640 1 -- 172.21.15.71:0/3680909018 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe5c137f60 con 0x7fbe5c142be0 2024-03-29T11:08:04.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.750+0000 7fbe58ff9640 1 -- 172.21.15.71:0/3680909018 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fbe50001870 con 0x7fbe5c142be0 2024-03-29T11:08:04.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.750+0000 7fbe58ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:04.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.750+0000 7fbe58ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:08:04.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.750+0000 7fbe58ff9640 10 monclient(hunting): dump: 2024-03-29T11:08:04.732 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:04.732 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:04.733 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:04.733 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:04.733 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:04.733 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:04.733 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:04.733 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:04.733 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:04.733 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:04.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.750+0000 7fbe58ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:08:04.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.752+0000 7fbe58ff9640 10 client.?.objecter ms_handle_connect 0x7fbe5c143d10 2024-03-29T11:08:04.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.752+0000 7fbe58ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:04.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.752+0000 7fbe58ff9640 1 -- 172.21.15.71:0/3680909018 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4036776022 0 0) 0x7fbe5c137f60 con 0x7fbe5c142be0 2024-03-29T11:08:04.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.752+0000 7fbe58ff9640 1 -- 172.21.15.71:0/3680909018 >> v1:172.21.15.71:6790/0 conn(0x7fbe5c0b1f60 legacy=0x7fbe5c131920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:08:04.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.754+0000 7fbe58ff9640 1 -- 172.21.15.71:0/3680909018 >> v1:172.21.15.137:6789/0 conn(0x7fbe5c143d10 legacy=0x7fbe5c137850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:04.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.754+0000 7fbe58ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:08:04.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.754+0000 7fbe58ff9640 1 monclient: found mon.a 2024-03-29T11:08:04.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.754+0000 7fbe58ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:04.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.754+0000 7fbe58ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:08:04.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.754+0000 7fbe58ff9640 1 -- 172.21.15.71:0/3680909018 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe5c1544b0 con 0x7fbe5c142be0 2024-03-29T11:08:04.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.754+0000 7fbe58ff9640 10 monclient: _finish_auth 0 2024-03-29T11:08:04.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.754+0000 7fbe58ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:08:04.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.754+0000 7fbe58ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:04.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.754+0000 7fbe62a4a640 5 monclient: authenticate success, global_id 4503 2024-03-29T11:08:04.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.754+0000 7fbe62a4a640 10 monclient: _renew_subs 2024-03-29T11:08:04.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.754+0000 7fbe62a4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:08:04.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.754+0000 7fbe62a4a640 1 -- 172.21.15.71:0/3680909018 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe5c154790 con 0x7fbe5c142be0 2024-03-29T11:08:04.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.754+0000 7fbe62a4a640 10 client.4503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:08:04.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.754+0000 7fbe62a4a640 10 monclient: _renew_subs 2024-03-29T11:08:04.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.754+0000 7fbe62a4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:08:04.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.754+0000 7fbe62a4a640 1 -- 172.21.15.71:0/3680909018 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe5c142190 con 0x7fbe5c142be0 2024-03-29T11:08:04.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.754+0000 7fbe62a4a640 1 librados: init done 2024-03-29T11:08:04.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.756+0000 7fbe58ff9640 1 -- 172.21.15.71:0/3680909018 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe500019d0 con 0x7fbe5c142be0 2024-03-29T11:08:04.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.756+0000 7fbe58ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:04.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.756+0000 7fbe62a4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:04.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.756+0000 7fbe58ff9640 1 -- 172.21.15.71:0/3680909018 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fbe500038f0 con 0x7fbe5c142be0 2024-03-29T11:08:04.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.756+0000 7fbe5a7fc640 4 set_mon_vals no callback set 2024-03-29T11:08:04.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.756+0000 7fbe62a4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:04.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.756+0000 7fbe62a4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:08:04.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.756+0000 7fbe62a4a640 1 -- 172.21.15.71:0/3680909018 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe5c154f50 con 0x7fbe5c142be0 2024-03-29T11:08:04.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.756+0000 7fbe58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:04.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.756+0000 7fbe58ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:08:04.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.756+0000 7fbe58ff9640 10 monclient: dump: 2024-03-29T11:08:04.738 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:04.738 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:04.738 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:04.739 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:04.739 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:04.739 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:04.739 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:04.739 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:04.739 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:04.739 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:04.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.759+0000 7fbe58ff9640 1 -- 172.21.15.71:0/3680909018 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7fbe50003a50 con 0x7fbe5c142be0 2024-03-29T11:08:04.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.760+0000 7fbe5bfff640 10 monclient: get_auth_request con 0x7fbe4806db20 auth_method 0 2024-03-29T11:08:04.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.761+0000 7fbe58ff9640 1 -- 172.21.15.71:0/3680909018 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7fbe50054750 con 0x7fbe5c142be0 2024-03-29T11:08:04.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.761+0000 7fbe58ff9640 10 client.4503.objecter ms_dispatch 0x7fbe5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:08:04.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.761+0000 7fbe58ff9640 3 client.4503.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:08:04.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.761+0000 7fbe58ff9640 3 client.4503.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:08:04.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.761+0000 7fbe58ff9640 20 client.4503.objecter dump_active .. 0 homeless 2024-03-29T11:08:04.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.761+0000 7fbe58ff9640 10 client.4503.objecter ms_handle_connect 0x7fbe4806db20 2024-03-29T11:08:04.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.768+0000 7fbe58ff9640 1 -- 172.21.15.71:0/3680909018 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbe50050090 con 0x7fbe5c142be0 2024-03-29T11:08:04.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.768+0000 7fbe58ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:04.746 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:04.768+0000 7fbe58ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:08:05.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:05.090+0000 7fbe62a4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:05.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:05.090+0000 7fbe62a4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:05.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:05.090+0000 7fbe62a4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:08:05.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:05.090+0000 7fbe62a4a640 1 -- 172.21.15.71:0/3680909018 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe5c075f10 con 0x7fbe5c142be0 2024-03-29T11:08:05.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:05.091+0000 7fbe58ff9640 1 -- 172.21.15.71:0/3680909018 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7fbe50052190 con 0x7fbe5c142be0 2024-03-29T11:08:05.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:05.091+0000 7fbe58ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:05.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:05.091+0000 7fbe58ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:08:05.070 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:08:05.070 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:08:05.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:05.094+0000 7fbe62a4a640 10 librados: watch_flush enter 2024-03-29T11:08:05.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:05.095+0000 7fbe62a4a640 10 librados: watch_flush exit 2024-03-29T11:08:05.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:05.095+0000 7fbe62a4a640 20 client.4503.objecter shutdown clearing up homeless session... 2024-03-29T11:08:05.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:05.095+0000 7fbe62a4a640 10 client.4503.objecter successfully canceled tick 2024-03-29T11:08:05.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:05.095+0000 7fbe62a4a640 1 -- 172.21.15.71:0/3680909018 >> v1:172.21.15.137:6816/973332665 conn(0x7fbe4806db20 legacy=0x7fbe4806ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:05.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:05.095+0000 7fbe62a4a640 10 monclient: shutdown 2024-03-29T11:08:05.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:05.095+0000 7fbe62a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:05.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:05.095+0000 7fbe62a4a640 1 -- 172.21.15.71:0/3680909018 >> v1:172.21.15.71:6789/0 conn(0x7fbe5c142be0 legacy=0x7fbe5c132030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:05.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:05.095+0000 7fbe62a4a640 1 -- 172.21.15.71:0/3680909018 shutdown_connections 2024-03-29T11:08:05.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:05.095+0000 7fbe62a4a640 1 -- 172.21.15.71:0/3680909018 >> 172.21.15.71:0/3680909018 conn(0x7fbe5c066a00 msgr2=0x7fbe5c0b4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:05.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:05.095+0000 7fbe62a4a640 1 -- 172.21.15.71:0/3680909018 shutdown_connections 2024-03-29T11:08:05.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:05.096+0000 7fbe62a4a640 1 -- 172.21.15.71:0/3680909018 wait complete. 2024-03-29T11:08:05.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:05.096+0000 7fbe62a4a640 1 librados: shutdown 2024-03-29T11:08:05.091 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:08:11.898 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:08:12.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.080+0000 7f457bc6e640 10 monclient: get_monmap_and_config 2024-03-29T11:08:12.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.080+0000 7f457bc6e640 10 monclient: build_initial_monmap 2024-03-29T11:08:12.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.080+0000 7f457bc6e640 10 monclient: monmap: 2024-03-29T11:08:12.059 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:08:12.060 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:12.060 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:08:12.080943+0000 2024-03-29T11:08:12.060 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:08:12.080943+0000 2024-03-29T11:08:12.060 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:12.060 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:12.060 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:08:12.060 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:08:12.060 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:08:12.060 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:12.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.081+0000 7f457bc6e640 1 Processor -- start 2024-03-29T11:08:12.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.081+0000 7f457bc6e640 1 -- start start 2024-03-29T11:08:12.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.081+0000 7f457bc6e640 10 monclient: init 2024-03-29T11:08:12.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.082+0000 7f457bc6e640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:12.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.082+0000 7f457bc6e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:08:12.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.082+0000 7f457bc6e640 10 monclient(hunting): picked mon.noname-c con 0x7f45740b0770 addr v1:172.21.15.137:6789/0 2024-03-29T11:08:12.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.082+0000 7f457bc6e640 10 monclient(hunting): picked mon.noname-b con 0x7f45740ad850 addr v1:172.21.15.71:6790/0 2024-03-29T11:08:12.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.082+0000 7f4579a0a640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f45740b0770 0x7f4574072070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:42938/0 (socket says 172.21.15.71:42938) 2024-03-29T11:08:12.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.082+0000 7f4579a0a640 1 -- 172.21.15.71:0/1626759424 learned_addr learned my addr 172.21.15.71:0/1626759424 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:08:12.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.082+0000 7f457bc6e640 10 monclient(hunting): picked mon.noname-a con 0x7f457413c1b0 addr v1:172.21.15.71:6789/0 2024-03-29T11:08:12.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.082+0000 7f457bc6e640 1 -- 172.21.15.71:0/1626759424 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45740ac8e0 con 0x7f457413c1b0 2024-03-29T11:08:12.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.083+0000 7f457bc6e640 1 -- 172.21.15.71:0/1626759424 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45740ad330 con 0x7f45740b0770 2024-03-29T11:08:12.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.083+0000 7f457bc6e640 1 -- 172.21.15.71:0/1626759424 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45740acfe0 con 0x7f45740ad850 2024-03-29T11:08:12.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.083+0000 7f457bc6e640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:12.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.083+0000 7f457bc6e640 10 monclient(hunting): authenticate will time out at 1248.341291s 2024-03-29T11:08:12.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.083+0000 7f4578a08640 1 -- 172.21.15.71:0/1626759424 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290397016 0 0) 0x7f45740ac8e0 con 0x7f457413c1b0 2024-03-29T11:08:12.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.083+0000 7f4578a08640 10 monclient(hunting): _negotiate 2024-03-29T11:08:12.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.083+0000 7f4578a08640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:12.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.083+0000 7f4578a08640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:12.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.083+0000 7f4578a08640 10 monclient(hunting): my global_id is 4506 2024-03-29T11:08:12.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.083+0000 7f4578a08640 1 -- 172.21.15.71:0/1626759424 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f455c005730 con 0x7f457413c1b0 2024-03-29T11:08:12.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.083+0000 7f4578a08640 1 -- 172.21.15.71:0/1626759424 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3354668913 0 0) 0x7f45740ad330 con 0x7f45740b0770 2024-03-29T11:08:12.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.083+0000 7f4578a08640 10 monclient(hunting): _negotiate 2024-03-29T11:08:12.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.083+0000 7f4578a08640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:12.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.083+0000 7f4578a08640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:12.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.083+0000 7f4578a08640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:08:12.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.083+0000 7f4578a08640 1 -- 172.21.15.71:0/1626759424 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45740ac8e0 con 0x7f45740b0770 2024-03-29T11:08:12.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.084+0000 7f4578a08640 1 -- 172.21.15.71:0/1626759424 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2828332275 0 0) 0x7f45740acfe0 con 0x7f45740ad850 2024-03-29T11:08:12.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.084+0000 7f4578a08640 10 monclient(hunting): _negotiate 2024-03-29T11:08:12.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.084+0000 7f4578a08640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:12.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.084+0000 7f4578a08640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:12.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.084+0000 7f4578a08640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:08:12.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.084+0000 7f4578a08640 1 -- 172.21.15.71:0/1626759424 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45740ad330 con 0x7f45740ad850 2024-03-29T11:08:12.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.084+0000 7f4578a08640 1 -- 172.21.15.71:0/1626759424 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1738123439 0 0) 0x7f455c005730 con 0x7f457413c1b0 2024-03-29T11:08:12.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.084+0000 7f4578a08640 1 -- 172.21.15.71:0/1626759424 >> v1:172.21.15.71:6790/0 conn(0x7f45740ad850 legacy=0x7f45740aaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:12.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.084+0000 7f4578a08640 1 -- 172.21.15.71:0/1626759424 >> v1:172.21.15.137:6789/0 conn(0x7f45740b0770 legacy=0x7f4574072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:12.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.084+0000 7f4578a08640 10 monclient: _finish_hunting 0 2024-03-29T11:08:12.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.084+0000 7f4578a08640 1 monclient: found mon.noname-a 2024-03-29T11:08:12.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.084+0000 7f4578a08640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:12.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.084+0000 7f4578a08640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:08:12.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.084+0000 7f4578a08640 1 -- 172.21.15.71:0/1626759424 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45740ab1f0 con 0x7f457413c1b0 2024-03-29T11:08:12.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.084+0000 7f4578a08640 10 monclient: _finish_auth 0 2024-03-29T11:08:12.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.084+0000 7f4578a08640 10 monclient: _check_auth_tickets 2024-03-29T11:08:12.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.084+0000 7f4578a08640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:12.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.084+0000 7f457bc6e640 5 monclient: authenticate success, global_id 4506 2024-03-29T11:08:12.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.084+0000 7f4578a08640 1 -- 172.21.15.71:0/1626759424 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4564001a10 con 0x7f457413c1b0 2024-03-29T11:08:12.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.084+0000 7f457bc6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:12.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.084+0000 7f4578a08640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:12.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.084+0000 7f4578a08640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:08:12.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.084+0000 7f4578a08640 10 monclient: dump: 2024-03-29T11:08:12.067 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:12.067 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:12.068 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:12.068 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:12.068 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:12.068 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:12.068 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:12.068 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:12.068 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:12.068 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:12.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.084+0000 7f457bc6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:12.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.084+0000 7f4578a08640 1 -- 172.21.15.71:0/1626759424 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4564001c30 con 0x7f457413c1b0 2024-03-29T11:08:12.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.084+0000 7f4578a08640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:12.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.085+0000 7f4578a08640 1 -- 172.21.15.71:0/1626759424 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4564001fc0 con 0x7f457413c1b0 2024-03-29T11:08:12.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.085+0000 7f4578a08640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:12.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.085+0000 7f4578a08640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:08:12.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.085+0000 7f4578a08640 10 monclient: dump: 2024-03-29T11:08:12.069 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:12.069 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:12.069 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:12.069 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:12.069 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:12.069 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:12.069 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:12.069 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:12.069 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:12.069 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:12.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.085+0000 7f457bc6e640 10 monclient: get_monmap_and_config success 2024-03-29T11:08:12.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.085+0000 7f457bc6e640 4 set_mon_vals no callback set 2024-03-29T11:08:12.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.086+0000 7f457bc6e640 10 monclient: shutdown 2024-03-29T11:08:12.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.086+0000 7f457bc6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:12.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.086+0000 7f457bc6e640 1 -- 172.21.15.71:0/1626759424 >> v1:172.21.15.71:6789/0 conn(0x7f457413c1b0 legacy=0x7f457414a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:12.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.086+0000 7f457bc6e640 1 -- 172.21.15.71:0/1626759424 shutdown_connections 2024-03-29T11:08:12.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.086+0000 7f457bc6e640 1 -- 172.21.15.71:0/1626759424 >> 172.21.15.71:0/1626759424 conn(0x7f457406b660 msgr2=0x7f457406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:12.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.086+0000 7f457bc6e640 1 -- 172.21.15.71:0/1626759424 shutdown_connections 2024-03-29T11:08:12.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.087+0000 7f457bc6e640 1 -- 172.21.15.71:0/1626759424 wait complete. 2024-03-29T11:08:12.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.088+0000 7f457bc6e640 10 monclient: build_initial_monmap 2024-03-29T11:08:12.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.088+0000 7f457bc6e640 10 monclient: monmap: 2024-03-29T11:08:12.077 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:08:12.077 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:08:12.077 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:08:12.077 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:08:12.078 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:12.078 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:12.078 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:08:12.078 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:08:12.078 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:08:12.078 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:12.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.089+0000 7f457bc6e640 1 librados: starting msgr at 2024-03-29T11:08:12.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.089+0000 7f457bc6e640 1 librados: starting objecter 2024-03-29T11:08:12.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.089+0000 7f457bc6e640 1 Processor -- start 2024-03-29T11:08:12.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.089+0000 7f457bc6e640 1 -- start start 2024-03-29T11:08:12.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.089+0000 7f457bc6e640 1 librados: setting wanted keys 2024-03-29T11:08:12.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.089+0000 7f457bc6e640 1 librados: calling monclient init 2024-03-29T11:08:12.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.089+0000 7f457bc6e640 10 monclient: init 2024-03-29T11:08:12.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.090+0000 7f457bc6e640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:12.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.090+0000 7f457bc6e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:08:12.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.090+0000 7f457bc6e640 10 monclient(hunting): picked mon.noname-c con 0x7f45740ad850 addr v1:172.21.15.71:6790/0 2024-03-29T11:08:12.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.090+0000 7f4579a0a640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f45740ad850 0x7f4574135740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:55338/0 (socket says 172.21.15.71:55338) 2024-03-29T11:08:12.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.090+0000 7f4579a0a640 1 -- 172.21.15.71:0/2516195286 learned_addr learned my addr 172.21.15.71:0/2516195286 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:08:12.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.091+0000 7f457bc6e640 10 monclient(hunting): picked mon.noname-a con 0x7f45740b0770 addr v1:172.21.15.71:6789/0 2024-03-29T11:08:12.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.091+0000 7f457bc6e640 10 monclient(hunting): picked mon.noname-b con 0x7f457413c1b0 addr v1:172.21.15.137:6789/0 2024-03-29T11:08:12.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.091+0000 7f457bc6e640 1 -- 172.21.15.71:0/2516195286 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457413f1a0 con 0x7f45740b0770 2024-03-29T11:08:12.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.091+0000 7f457bc6e640 1 -- 172.21.15.71:0/2516195286 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457413ee40 con 0x7f457413c1b0 2024-03-29T11:08:12.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.091+0000 7f457bc6e640 1 -- 172.21.15.71:0/2516195286 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457413e510 con 0x7f45740ad850 2024-03-29T11:08:12.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.091+0000 7f457bc6e640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:12.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.091+0000 7f457bc6e640 10 monclient(hunting): authenticate will time out at 1248.349699s 2024-03-29T11:08:12.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.091+0000 7f45727fc640 10 client.?.objecter ms_handle_connect 0x7f45740ad850 2024-03-29T11:08:12.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.091+0000 7f45727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:12.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.091+0000 7f45727fc640 10 client.?.objecter ms_handle_connect 0x7f457413c1b0 2024-03-29T11:08:12.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.091+0000 7f45727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:12.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.091+0000 7f45727fc640 10 client.?.objecter ms_handle_connect 0x7f45740b0770 2024-03-29T11:08:12.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.091+0000 7f45727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:12.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.091+0000 7f45727fc640 1 -- 172.21.15.71:0/2516195286 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3917815187 0 0) 0x7f457413f1a0 con 0x7f45740b0770 2024-03-29T11:08:12.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.091+0000 7f45727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:08:12.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.091+0000 7f45727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:12.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.092+0000 7f45727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:12.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.092+0000 7f45727fc640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:08:12.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.092+0000 7f45727fc640 1 -- 172.21.15.71:0/2516195286 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4560004eb0 con 0x7f45740b0770 2024-03-29T11:08:12.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.092+0000 7f45727fc640 1 -- 172.21.15.71:0/2516195286 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2239753249 0 0) 0x7f457413ee40 con 0x7f457413c1b0 2024-03-29T11:08:12.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.092+0000 7f45727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:08:12.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.092+0000 7f45727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:12.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.092+0000 7f45727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:12.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.092+0000 7f45727fc640 10 monclient(hunting): my global_id is 4462 2024-03-29T11:08:12.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.092+0000 7f45727fc640 1 -- 172.21.15.71:0/2516195286 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f457413f1a0 con 0x7f457413c1b0 2024-03-29T11:08:12.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.092+0000 7f45727fc640 1 -- 172.21.15.71:0/2516195286 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318870669 0 0) 0x7f457413e510 con 0x7f45740ad850 2024-03-29T11:08:12.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.092+0000 7f45727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:08:12.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.092+0000 7f45727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:12.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.092+0000 7f45727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:12.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.092+0000 7f45727fc640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:08:12.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.092+0000 7f45727fc640 1 -- 172.21.15.71:0/2516195286 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f457413ee40 con 0x7f45740ad850 2024-03-29T11:08:12.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.092+0000 7f45727fc640 1 -- 172.21.15.71:0/2516195286 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 306543587 0 0) 0x7f4560004eb0 con 0x7f45740b0770 2024-03-29T11:08:12.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.092+0000 7f45727fc640 1 -- 172.21.15.71:0/2516195286 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f457413e510 con 0x7f45740b0770 2024-03-29T11:08:12.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.092+0000 7f45727fc640 1 -- 172.21.15.71:0/2516195286 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f45680018f0 con 0x7f45740b0770 2024-03-29T11:08:12.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.092+0000 7f45727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:12.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.092+0000 7f45727fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:08:12.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.092+0000 7f45727fc640 10 monclient(hunting): dump: 2024-03-29T11:08:12.082 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:12.082 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:12.082 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:12.082 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:12.082 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:12.082 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:12.082 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:12.082 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:12.083 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:12.083 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:12.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.092+0000 7f45727fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:08:12.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.092+0000 7f45727fc640 1 -- 172.21.15.71:0/2516195286 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1122553810 0 0) 0x7f457413ee40 con 0x7f45740ad850 2024-03-29T11:08:12.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f45727fc640 1 -- 172.21.15.71:0/2516195286 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4560004eb0 con 0x7f45740ad850 2024-03-29T11:08:12.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f45727fc640 1 -- 172.21.15.71:0/2516195286 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f456c0018f0 con 0x7f45740ad850 2024-03-29T11:08:12.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f45727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:12.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f45727fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:12.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f45727fc640 10 monclient(hunting): dump: 2024-03-29T11:08:12.083 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:12.083 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:12.083 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:12.083 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:12.083 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:12.083 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:12.083 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:12.084 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:12.084 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:12.084 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:12.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f45727fc640 1 -- 172.21.15.71:0/2516195286 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 63510035 0 0) 0x7f457413f1a0 con 0x7f457413c1b0 2024-03-29T11:08:12.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f45727fc640 1 -- 172.21.15.71:0/2516195286 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f457413ee40 con 0x7f457413c1b0 2024-03-29T11:08:12.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f45727fc640 1 -- 172.21.15.71:0/2516195286 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4564001da0 con 0x7f457413c1b0 2024-03-29T11:08:12.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f45727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:12.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f45727fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:08:12.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f45727fc640 10 monclient(hunting): dump: 2024-03-29T11:08:12.084 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:12.084 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:12.084 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:12.084 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:12.084 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:12.084 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:12.085 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:12.085 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:12.085 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:12.085 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:12.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f45727fc640 1 -- 172.21.15.71:0/2516195286 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1372948654 0 0) 0x7f457413e510 con 0x7f45740b0770 2024-03-29T11:08:12.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f45727fc640 1 -- 172.21.15.71:0/2516195286 >> v1:172.21.15.71:6790/0 conn(0x7f45740ad850 legacy=0x7f4574135740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:12.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f45727fc640 1 -- 172.21.15.71:0/2516195286 >> v1:172.21.15.137:6789/0 conn(0x7f457413c1b0 legacy=0x7f45741e6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:12.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f45727fc640 10 monclient: _finish_hunting 0 2024-03-29T11:08:12.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f45727fc640 1 monclient: found mon.a 2024-03-29T11:08:12.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f45727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:12.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f45727fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:08:12.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f45727fc640 1 -- 172.21.15.71:0/2516195286 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457413e6a0 con 0x7f45740b0770 2024-03-29T11:08:12.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f45727fc640 10 monclient: _finish_auth 0 2024-03-29T11:08:12.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f45727fc640 10 monclient: _check_auth_tickets 2024-03-29T11:08:12.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f45727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:12.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f457bc6e640 5 monclient: authenticate success, global_id 4509 2024-03-29T11:08:12.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f457bc6e640 10 monclient: _renew_subs 2024-03-29T11:08:12.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f457bc6e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:08:12.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f457bc6e640 1 -- 172.21.15.71:0/2516195286 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f457413e050 con 0x7f45740b0770 2024-03-29T11:08:12.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f457bc6e640 10 client.4509.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:08:12.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f457bc6e640 10 monclient: _renew_subs 2024-03-29T11:08:12.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f457bc6e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:08:12.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f457bc6e640 1 -- 172.21.15.71:0/2516195286 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45740b2490 con 0x7f45740b0770 2024-03-29T11:08:12.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.093+0000 7f457bc6e640 1 librados: init done 2024-03-29T11:08:12.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.094+0000 7f45727fc640 1 -- 172.21.15.71:0/2516195286 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4568003610 con 0x7f45740b0770 2024-03-29T11:08:12.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.094+0000 7f45727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:12.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.094+0000 7f45727fc640 1 -- 172.21.15.71:0/2516195286 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4568003920 con 0x7f45740b0770 2024-03-29T11:08:12.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.094+0000 7f45727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:12.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.094+0000 7f45727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:08:12.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.094+0000 7f45727fc640 10 monclient: dump: 2024-03-29T11:08:12.086 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:12.087 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:12.087 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:12.087 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:12.087 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:12.087 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:12.087 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:12.087 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:12.087 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:12.087 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:12.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.094+0000 7f45737fe640 4 set_mon_vals no callback set 2024-03-29T11:08:12.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.094+0000 7f457bc6e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:12.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.095+0000 7f457bc6e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:12.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.095+0000 7f457bc6e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:08:12.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.095+0000 7f457bc6e640 1 -- 172.21.15.71:0/2516195286 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f457413efd0 con 0x7f45740b0770 2024-03-29T11:08:12.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.096+0000 7f45727fc640 1 -- 172.21.15.71:0/2516195286 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f45680198f0 con 0x7f45740b0770 2024-03-29T11:08:12.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.096+0000 7f45727fc640 1 -- 172.21.15.71:0/2516195286 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f4568083800 con 0x7f45740b0770 2024-03-29T11:08:12.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.096+0000 7f45727fc640 10 client.4509.objecter ms_dispatch 0x7f4574000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:08:12.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.096+0000 7f45727fc640 3 client.4509.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:08:12.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.096+0000 7f45727fc640 3 client.4509.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:08:12.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.097+0000 7f45727fc640 20 client.4509.objecter dump_active .. 0 homeless 2024-03-29T11:08:12.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.097+0000 7f4579a0a640 10 monclient: get_auth_request con 0x7f456006e190 auth_method 0 2024-03-29T11:08:12.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.097+0000 7f45727fc640 10 client.4509.objecter ms_handle_connect 0x7f456006e190 2024-03-29T11:08:12.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.108+0000 7f45727fc640 1 -- 172.21.15.71:0/2516195286 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f456804ef10 con 0x7f45740b0770 2024-03-29T11:08:12.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.108+0000 7f45727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:12.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.108+0000 7f45727fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:08:12.447 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.469+0000 7f457bc6e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:12.447 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.469+0000 7f457bc6e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:12.448 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.469+0000 7f457bc6e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:08:12.448 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.469+0000 7f457bc6e640 1 -- 172.21.15.71:0/2516195286 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4574075f10 con 0x7f45740b0770 2024-03-29T11:08:12.448 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.470+0000 7f45727fc640 1 -- 172.21.15.71:0/2516195286 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f4568051010 con 0x7f45740b0770 2024-03-29T11:08:12.448 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.470+0000 7f45727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:12.448 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.470+0000 7f45727fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:08:12.448 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:08:12.448 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:08:12.451 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.473+0000 7f457bc6e640 10 librados: watch_flush enter 2024-03-29T11:08:12.451 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.473+0000 7f457bc6e640 10 librados: watch_flush exit 2024-03-29T11:08:12.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.473+0000 7f457bc6e640 20 client.4509.objecter shutdown clearing up homeless session... 2024-03-29T11:08:12.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.473+0000 7f457bc6e640 10 client.4509.objecter successfully canceled tick 2024-03-29T11:08:12.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.473+0000 7f457bc6e640 1 -- 172.21.15.71:0/2516195286 >> v1:172.21.15.137:6816/973332665 conn(0x7f456006e190 legacy=0x7f4560070650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:12.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.473+0000 7f457bc6e640 10 monclient: shutdown 2024-03-29T11:08:12.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.473+0000 7f457bc6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:12.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.473+0000 7f457bc6e640 1 -- 172.21.15.71:0/2516195286 >> v1:172.21.15.71:6789/0 conn(0x7f45740b0770 legacy=0x7f45741debf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:12.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.473+0000 7f457bc6e640 1 -- 172.21.15.71:0/2516195286 shutdown_connections 2024-03-29T11:08:12.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.473+0000 7f457bc6e640 1 -- 172.21.15.71:0/2516195286 >> 172.21.15.71:0/2516195286 conn(0x7f457406b660 msgr2=0x7f45740a9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:12.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.473+0000 7f457bc6e640 1 -- 172.21.15.71:0/2516195286 shutdown_connections 2024-03-29T11:08:12.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.473+0000 7f457bc6e640 1 -- 172.21.15.71:0/2516195286 wait complete. 2024-03-29T11:08:12.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:12.473+0000 7f457bc6e640 1 librados: shutdown 2024-03-29T11:08:12.467 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:08:19.274 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:08:19.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.445+0000 7f8d5d17c640 10 monclient: get_monmap_and_config 2024-03-29T11:08:19.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.445+0000 7f8d5d17c640 10 monclient: build_initial_monmap 2024-03-29T11:08:19.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.445+0000 7f8d5d17c640 10 monclient: monmap: 2024-03-29T11:08:19.424 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:08:19.424 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:19.424 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:08:19.445965+0000 2024-03-29T11:08:19.424 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:08:19.445965+0000 2024-03-29T11:08:19.424 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:19.424 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:19.424 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:08:19.424 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:08:19.424 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:08:19.424 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:19.425 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.446+0000 7f8d5d17c640 1 Processor -- start 2024-03-29T11:08:19.425 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.446+0000 7f8d5d17c640 1 -- start start 2024-03-29T11:08:19.425 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.446+0000 7f8d5d17c640 10 monclient: init 2024-03-29T11:08:19.426 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.447+0000 7f8d5d17c640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:19.426 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.447+0000 7f8d5d17c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:08:19.426 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.447+0000 7f8d5d17c640 10 monclient(hunting): picked mon.noname-b con 0x7f8d58149c50 addr v1:172.21.15.71:6790/0 2024-03-29T11:08:19.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.447+0000 7f8d5d17c640 10 monclient(hunting): picked mon.noname-c con 0x7f8d580a9cc0 addr v1:172.21.15.137:6789/0 2024-03-29T11:08:19.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.447+0000 7f8d5d17c640 10 monclient(hunting): picked mon.noname-a con 0x7f8d58143d00 addr v1:172.21.15.71:6789/0 2024-03-29T11:08:19.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.447+0000 7f8d5d17c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d580ab9c0 con 0x7f8d58143d00 2024-03-29T11:08:19.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.447+0000 7f8d5d17c640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d580abc10 con 0x7f8d580a9cc0 2024-03-29T11:08:19.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.447+0000 7f8d5d17c640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d580aad20 con 0x7f8d58149c50 2024-03-29T11:08:19.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.447+0000 7f8d5d17c640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:19.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.447+0000 7f8d5d17c640 10 monclient(hunting): authenticate will time out at 1255.705927s 2024-03-29T11:08:19.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.448+0000 7f8d5759e640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f8d58143d00 0x7f8d58151d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:48280/0 (socket says 172.21.15.71:48280) 2024-03-29T11:08:19.427 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.448+0000 7f8d5759e640 1 -- 172.21.15.71:0/3645389286 learned_addr learned my addr 172.21.15.71:0/3645389286 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:08:19.428 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.448+0000 7f8d55d9b640 1 -- 172.21.15.71:0/3645389286 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442364207 0 0) 0x7f8d580ab9c0 con 0x7f8d58143d00 2024-03-29T11:08:19.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.448+0000 7f8d55d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:08:19.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.448+0000 7f8d55d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:19.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.448+0000 7f8d55d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:19.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.448+0000 7f8d55d9b640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:08:19.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 1 -- 172.21.15.71:0/3645389286 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d40003520 con 0x7f8d58143d00 2024-03-29T11:08:19.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 1 -- 172.21.15.71:0/3645389286 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288920024 0 0) 0x7f8d580aad20 con 0x7f8d58149c50 2024-03-29T11:08:19.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:08:19.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:19.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:19.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:08:19.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 1 -- 172.21.15.71:0/3645389286 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d580ab9c0 con 0x7f8d58149c50 2024-03-29T11:08:19.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 1 -- 172.21.15.71:0/3645389286 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623012479 0 0) 0x7f8d580abc10 con 0x7f8d580a9cc0 2024-03-29T11:08:19.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:08:19.429 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:19.433 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:19.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:08:19.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 1 -- 172.21.15.71:0/3645389286 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d580aad20 con 0x7f8d580a9cc0 2024-03-29T11:08:19.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 1 -- 172.21.15.71:0/3645389286 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1786742205 0 0) 0x7f8d40003520 con 0x7f8d58143d00 2024-03-29T11:08:19.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 1 -- 172.21.15.71:0/3645389286 >> v1:172.21.15.71:6790/0 conn(0x7f8d58149c50 legacy=0x7f8d580ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:19.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 1 -- 172.21.15.71:0/3645389286 >> v1:172.21.15.137:6789/0 conn(0x7f8d580a9cc0 legacy=0x7f8d581435f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:19.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:08:19.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 1 monclient: found mon.noname-a 2024-03-29T11:08:19.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:19.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:08:19.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 1 -- 172.21.15.71:0/3645389286 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d580aa980 con 0x7f8d58143d00 2024-03-29T11:08:19.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 10 monclient: _finish_auth 0 2024-03-29T11:08:19.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:08:19.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:19.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 1 -- 172.21.15.71:0/3645389286 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f8d48001610 con 0x7f8d58143d00 2024-03-29T11:08:19.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:19.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:08:19.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 10 monclient: dump: 2024-03-29T11:08:19.435 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:19.435 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:19.435 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:19.435 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:19.435 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:19.435 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:19.435 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:19.435 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:19.435 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:19.435 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:19.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d5d17c640 5 monclient: authenticate success, global_id 4512 2024-03-29T11:08:19.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d5d17c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:19.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 1 -- 172.21.15.71:0/3645389286 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d48001830 con 0x7f8d58143d00 2024-03-29T11:08:19.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:19.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 1 -- 172.21.15.71:0/3645389286 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f8d48001bc0 con 0x7f8d58143d00 2024-03-29T11:08:19.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:19.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:08:19.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d55d9b640 10 monclient: dump: 2024-03-29T11:08:19.436 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:19.438 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:19.438 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:19.438 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:19.438 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:19.438 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:19.438 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:19.438 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:19.438 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:19.438 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:19.439 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d5d17c640 10 monclient: get_monmap_and_config success 2024-03-29T11:08:19.439 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.449+0000 7f8d5d17c640 4 set_mon_vals no callback set 2024-03-29T11:08:19.439 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.450+0000 7f8d5d17c640 10 monclient: shutdown 2024-03-29T11:08:19.439 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.450+0000 7f8d5d17c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:19.439 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.450+0000 7f8d5d17c640 1 -- 172.21.15.71:0/3645389286 >> v1:172.21.15.71:6789/0 conn(0x7f8d58143d00 legacy=0x7f8d58151d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:19.439 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.450+0000 7f8d5d17c640 1 -- 172.21.15.71:0/3645389286 shutdown_connections 2024-03-29T11:08:19.439 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.450+0000 7f8d5d17c640 1 -- 172.21.15.71:0/3645389286 >> 172.21.15.71:0/3645389286 conn(0x7f8d580b3c50 msgr2=0x7f8d580b4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:19.439 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.450+0000 7f8d5d17c640 1 -- 172.21.15.71:0/3645389286 shutdown_connections 2024-03-29T11:08:19.439 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.450+0000 7f8d5d17c640 1 -- 172.21.15.71:0/3645389286 wait complete. 2024-03-29T11:08:19.439 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.452+0000 7f8d5d17c640 10 monclient: build_initial_monmap 2024-03-29T11:08:19.439 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.452+0000 7f8d5d17c640 10 monclient: monmap: 2024-03-29T11:08:19.439 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:08:19.439 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:08:19.439 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:08:19.439 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:08:19.439 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:19.440 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:19.440 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:08:19.440 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:08:19.440 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:08:19.440 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:19.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.452+0000 7f8d5d17c640 1 librados: starting msgr at 2024-03-29T11:08:19.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.452+0000 7f8d5d17c640 1 librados: starting objecter 2024-03-29T11:08:19.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.453+0000 7f8d5d17c640 1 Processor -- start 2024-03-29T11:08:19.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.453+0000 7f8d5d17c640 1 -- start start 2024-03-29T11:08:19.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.453+0000 7f8d5d17c640 1 librados: setting wanted keys 2024-03-29T11:08:19.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.453+0000 7f8d5d17c640 1 librados: calling monclient init 2024-03-29T11:08:19.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.453+0000 7f8d5d17c640 10 monclient: init 2024-03-29T11:08:19.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.454+0000 7f8d5d17c640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:19.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.454+0000 7f8d5d17c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:08:19.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.454+0000 7f8d5d17c640 10 monclient(hunting): picked mon.noname-b con 0x7f8d580a9cc0 addr v1:172.21.15.137:6789/0 2024-03-29T11:08:19.441 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.454+0000 7f8d5d17c640 10 monclient(hunting): picked mon.noname-a con 0x7f8d58143d00 addr v1:172.21.15.71:6789/0 2024-03-29T11:08:19.441 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.454+0000 7f8d5d17c640 10 monclient(hunting): picked mon.noname-c con 0x7f8d58149c50 addr v1:172.21.15.71:6790/0 2024-03-29T11:08:19.441 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.454+0000 7f8d5d17c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d5806e6e0 con 0x7f8d58143d00 2024-03-29T11:08:19.449 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.454+0000 7f8d5d17c640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d581450f0 con 0x7f8d580a9cc0 2024-03-29T11:08:19.449 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.454+0000 7f8d5d17c640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d58070a90 con 0x7f8d58149c50 2024-03-29T11:08:19.449 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.454+0000 7f8d5d17c640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:19.449 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.454+0000 7f8d5d17c640 10 monclient(hunting): authenticate will time out at 1255.712235s 2024-03-29T11:08:19.450 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.454+0000 7f8d56d9d640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f8d580a9cc0 0x7f8d5806eea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:60306/0 (socket says 172.21.15.71:60306) 2024-03-29T11:08:19.450 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.454+0000 7f8d56d9d640 1 -- 172.21.15.71:0/3618756640 learned_addr learned my addr 172.21.15.71:0/3618756640 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:08:19.450 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.454+0000 7f8d3f7fe640 10 client.?.objecter ms_handle_connect 0x7f8d580a9cc0 2024-03-29T11:08:19.450 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.454+0000 7f8d3f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:19.450 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.454+0000 7f8d3f7fe640 10 client.?.objecter ms_handle_connect 0x7f8d58149c50 2024-03-29T11:08:19.450 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.454+0000 7f8d3f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:19.450 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.454+0000 7f8d3f7fe640 1 -- 172.21.15.71:0/3618756640 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095217386 0 0) 0x7f8d581450f0 con 0x7f8d580a9cc0 2024-03-29T11:08:19.450 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.454+0000 7f8d3f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:08:19.450 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.454+0000 7f8d3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:19.451 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.454+0000 7f8d3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:19.451 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.454+0000 7f8d3f7fe640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:08:19.451 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.454+0000 7f8d3f7fe640 1 -- 172.21.15.71:0/3618756640 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d44004eb0 con 0x7f8d580a9cc0 2024-03-29T11:08:19.451 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.455+0000 7f8d3f7fe640 10 client.?.objecter ms_handle_connect 0x7f8d58143d00 2024-03-29T11:08:19.451 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.455+0000 7f8d3f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:19.451 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.455+0000 7f8d3f7fe640 1 -- 172.21.15.71:0/3618756640 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 649100432 0 0) 0x7f8d44004eb0 con 0x7f8d580a9cc0 2024-03-29T11:08:19.451 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.455+0000 7f8d3f7fe640 1 -- 172.21.15.71:0/3618756640 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d581450f0 con 0x7f8d580a9cc0 2024-03-29T11:08:19.451 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.455+0000 7f8d3f7fe640 1 -- 172.21.15.71:0/3618756640 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f8d50001cf0 con 0x7f8d580a9cc0 2024-03-29T11:08:19.451 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.455+0000 7f8d3f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:19.451 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.455+0000 7f8d3f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:08:19.451 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.455+0000 7f8d3f7fe640 10 monclient(hunting): dump: 2024-03-29T11:08:19.452 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:19.452 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:19.452 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:19.452 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:19.452 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:19.452 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:19.452 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:19.452 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:19.452 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:19.452 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:19.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.455+0000 7f8d3f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:08:19.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.455+0000 7f8d3f7fe640 1 -- 172.21.15.71:0/3618756640 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3701775578 0 0) 0x7f8d58070a90 con 0x7f8d58149c50 2024-03-29T11:08:19.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.455+0000 7f8d3f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:08:19.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.455+0000 7f8d3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:19.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.455+0000 7f8d3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:19.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.455+0000 7f8d3f7fe640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:08:19.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.455+0000 7f8d3f7fe640 1 -- 172.21.15.71:0/3618756640 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d44004eb0 con 0x7f8d58149c50 2024-03-29T11:08:19.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.455+0000 7f8d3f7fe640 1 -- 172.21.15.71:0/3618756640 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2391097720 0 0) 0x7f8d5806e6e0 con 0x7f8d58143d00 2024-03-29T11:08:19.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.455+0000 7f8d3f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:08:19.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.455+0000 7f8d3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:19.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.455+0000 7f8d3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:19.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.455+0000 7f8d3f7fe640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:08:19.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.455+0000 7f8d3f7fe640 1 -- 172.21.15.71:0/3618756640 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d58070a90 con 0x7f8d58143d00 2024-03-29T11:08:19.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d3f7fe640 1 -- 172.21.15.71:0/3618756640 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2229757358 0 0) 0x7f8d44004eb0 con 0x7f8d58149c50 2024-03-29T11:08:19.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d3f7fe640 1 -- 172.21.15.71:0/3618756640 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d5806e6e0 con 0x7f8d58149c50 2024-03-29T11:08:19.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d3f7fe640 1 -- 172.21.15.71:0/3618756640 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f8d48001550 con 0x7f8d58149c50 2024-03-29T11:08:19.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d3f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:19.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d3f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:19.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d3f7fe640 10 monclient(hunting): dump: 2024-03-29T11:08:19.454 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:19.454 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:19.454 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:19.455 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:19.455 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:19.455 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:19.455 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:19.455 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:19.455 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:19.455 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:19.455 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d3f7fe640 1 -- 172.21.15.71:0/3618756640 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2445436295 0 0) 0x7f8d581450f0 con 0x7f8d580a9cc0 2024-03-29T11:08:19.455 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d3f7fe640 1 -- 172.21.15.71:0/3618756640 >> v1:172.21.15.71:6790/0 conn(0x7f8d58149c50 legacy=0x7f8d581e6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:19.455 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d3f7fe640 1 -- 172.21.15.71:0/3618756640 >> v1:172.21.15.71:6789/0 conn(0x7f8d58143d00 legacy=0x7f8d580721f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:19.455 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d3f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:08:19.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d3f7fe640 1 monclient: found mon.b 2024-03-29T11:08:19.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:19.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d3f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:08:19.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d3f7fe640 1 -- 172.21.15.71:0/3618756640 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5806e980 con 0x7f8d580a9cc0 2024-03-29T11:08:19.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d3f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:08:19.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d3f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:08:19.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:19.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d5d17c640 5 monclient: authenticate success, global_id 4468 2024-03-29T11:08:19.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d5d17c640 10 monclient: _renew_subs 2024-03-29T11:08:19.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d5d17c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:08:19.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d5d17c640 1 -- 172.21.15.71:0/3618756640 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d5806e430 con 0x7f8d580a9cc0 2024-03-29T11:08:19.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d5d17c640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:08:19.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d5d17c640 10 monclient: _renew_subs 2024-03-29T11:08:19.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d5d17c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:08:19.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d5d17c640 1 -- 172.21.15.71:0/3618756640 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d581dc430 con 0x7f8d580a9cc0 2024-03-29T11:08:19.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d5d17c640 1 librados: init done 2024-03-29T11:08:19.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d3f7fe640 1 -- 172.21.15.71:0/3618756640 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d50003990 con 0x7f8d580a9cc0 2024-03-29T11:08:19.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:19.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d3f7fe640 1 -- 172.21.15.71:0/3618756640 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f8d50003da0 con 0x7f8d580a9cc0 2024-03-29T11:08:19.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:19.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d3f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:08:19.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d3f7fe640 10 monclient: dump: 2024-03-29T11:08:19.458 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:19.458 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:19.458 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:19.458 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:19.458 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:19.458 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:19.458 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:19.458 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:19.458 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:19.458 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:19.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.456+0000 7f8d5559a640 4 set_mon_vals no callback set 2024-03-29T11:08:19.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.457+0000 7f8d5d17c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:19.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.457+0000 7f8d5d17c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:19.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.458+0000 7f8d5d17c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:08:19.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.458+0000 7f8d5d17c640 1 -- 172.21.15.71:0/3618756640 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d38005080 con 0x7f8d580a9cc0 2024-03-29T11:08:19.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.458+0000 7f8d3f7fe640 1 -- 172.21.15.71:0/3618756640 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f8d50003f00 con 0x7f8d580a9cc0 2024-03-29T11:08:19.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.459+0000 7f8d3f7fe640 1 -- 172.21.15.71:0/3618756640 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f8d50084e50 con 0x7f8d580a9cc0 2024-03-29T11:08:19.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.459+0000 7f8d3f7fe640 10 client.4468.objecter ms_dispatch 0x7f8d58000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:08:19.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.459+0000 7f8d3f7fe640 3 client.4468.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:08:19.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.459+0000 7f8d3f7fe640 3 client.4468.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:08:19.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.459+0000 7f8d3f7fe640 20 client.4468.objecter dump_active .. 0 homeless 2024-03-29T11:08:19.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.459+0000 7f8d5659c640 10 monclient: get_auth_request con 0x7f8d4406e000 auth_method 0 2024-03-29T11:08:19.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.460+0000 7f8d3f7fe640 10 client.4468.objecter ms_handle_connect 0x7f8d4406e000 2024-03-29T11:08:19.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.468+0000 7f8d3f7fe640 1 -- 172.21.15.71:0/3618756640 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8d50050730 con 0x7f8d580a9cc0 2024-03-29T11:08:19.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.468+0000 7f8d3f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:19.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.468+0000 7f8d3f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:08:19.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.822+0000 7f8d5d17c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:19.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.822+0000 7f8d5d17c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:19.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.822+0000 7f8d5d17c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:08:19.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.822+0000 7f8d5d17c640 1 -- 172.21.15.71:0/3618756640 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d38005330 con 0x7f8d580a9cc0 2024-03-29T11:08:19.806 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.824+0000 7f8d3f7fe640 1 -- 172.21.15.71:0/3618756640 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f8d50052660 con 0x7f8d580a9cc0 2024-03-29T11:08:19.806 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.824+0000 7f8d3f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:19.806 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.824+0000 7f8d3f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:08:19.807 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:08:19.807 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:08:19.807 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.828+0000 7f8d5d17c640 10 librados: watch_flush enter 2024-03-29T11:08:19.807 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.828+0000 7f8d5d17c640 10 librados: watch_flush exit 2024-03-29T11:08:19.807 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.828+0000 7f8d5d17c640 20 client.4468.objecter shutdown clearing up homeless session... 2024-03-29T11:08:19.807 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.828+0000 7f8d5d17c640 10 client.4468.objecter successfully canceled tick 2024-03-29T11:08:19.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.828+0000 7f8d5d17c640 1 -- 172.21.15.71:0/3618756640 >> v1:172.21.15.137:6816/973332665 conn(0x7f8d4406e000 legacy=0x7f8d440704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:19.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.828+0000 7f8d5d17c640 10 monclient: shutdown 2024-03-29T11:08:19.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.828+0000 7f8d5d17c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:19.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.828+0000 7f8d5d17c640 1 -- 172.21.15.71:0/3618756640 >> v1:172.21.15.137:6789/0 conn(0x7f8d580a9cc0 legacy=0x7f8d5806eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:19.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.828+0000 7f8d5d17c640 1 -- 172.21.15.71:0/3618756640 shutdown_connections 2024-03-29T11:08:19.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.828+0000 7f8d5d17c640 1 -- 172.21.15.71:0/3618756640 >> 172.21.15.71:0/3618756640 conn(0x7f8d580b3c50 msgr2=0x7f8d581433c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:19.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.828+0000 7f8d5d17c640 1 -- 172.21.15.71:0/3618756640 shutdown_connections 2024-03-29T11:08:19.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.828+0000 7f8d5d17c640 1 -- 172.21.15.71:0/3618756640 wait complete. 2024-03-29T11:08:19.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:19.828+0000 7f8d5d17c640 1 librados: shutdown 2024-03-29T11:08:19.823 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:08:26.630 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:08:26.787 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.807+0000 7f1fc50c9640 10 monclient: get_monmap_and_config 2024-03-29T11:08:26.787 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.807+0000 7f1fc50c9640 10 monclient: build_initial_monmap 2024-03-29T11:08:26.787 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.807+0000 7f1fc50c9640 10 monclient: monmap: 2024-03-29T11:08:26.787 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:08:26.787 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:26.788 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:08:26.808564+0000 2024-03-29T11:08:26.788 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:08:26.808564+0000 2024-03-29T11:08:26.788 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:26.788 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:26.788 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:08:26.788 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:08:26.788 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:08:26.788 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:26.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.811+0000 7f1fc50c9640 1 Processor -- start 2024-03-29T11:08:26.790 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.811+0000 7f1fc50c9640 1 -- start start 2024-03-29T11:08:26.790 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.811+0000 7f1fc50c9640 10 monclient: init 2024-03-29T11:08:26.791 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.812+0000 7f1fc50c9640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:26.791 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.812+0000 7f1fc50c9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:08:26.791 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.812+0000 7f1fc50c9640 10 monclient(hunting): picked mon.noname-b con 0x7f1fb8034850 addr v1:172.21.15.71:6790/0 2024-03-29T11:08:26.791 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.812+0000 7f1fc50c9640 10 monclient(hunting): picked mon.noname-a con 0x7f1fb802a640 addr v1:172.21.15.71:6789/0 2024-03-29T11:08:26.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.812+0000 7f1fc50c9640 10 monclient(hunting): picked mon.noname-c con 0x7f1fb80327d0 addr v1:172.21.15.137:6789/0 2024-03-29T11:08:26.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.812+0000 7f1fc50c9640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb802c820 con 0x7f1fb802a640 2024-03-29T11:08:26.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.812+0000 7f1fc50c9640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb802a120 con 0x7f1fb80327d0 2024-03-29T11:08:26.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.812+0000 7f1fc50c9640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb80c75b0 con 0x7f1fb8034850 2024-03-29T11:08:26.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.812+0000 7f1fc50c9640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:26.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.812+0000 7f1fc50c9640 10 monclient(hunting): authenticate will time out at 1263.070365s 2024-03-29T11:08:26.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.816+0000 7f1fbffff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f1fb8034850 0x7f1fb80c6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:48353/0 (socket says 172.21.15.71:48353) 2024-03-29T11:08:26.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.816+0000 7f1fbffff640 1 -- 172.21.15.71:0/688343682 learned_addr learned my addr 172.21.15.71:0/688343682 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:08:26.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.818+0000 7f1fbeffd640 1 -- 172.21.15.71:0/688343682 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290241198 0 0) 0x7f1fb80c75b0 con 0x7f1fb8034850 2024-03-29T11:08:26.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.818+0000 7f1fbeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:08:26.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.818+0000 7f1fbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:26.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.818+0000 7f1fbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:26.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.818+0000 7f1fbeffd640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:08:26.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.818+0000 7f1fbeffd640 1 -- 172.21.15.71:0/688343682 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fb0003520 con 0x7f1fb8034850 2024-03-29T11:08:26.799 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fbeffd640 1 -- 172.21.15.71:0/688343682 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2532369764 0 0) 0x7f1fb802a120 con 0x7f1fb80327d0 2024-03-29T11:08:26.799 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fbeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:08:26.799 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:26.799 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:26.799 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fbeffd640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:08:26.799 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fbeffd640 1 -- 172.21.15.71:0/688343682 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fb80c75b0 con 0x7f1fb80327d0 2024-03-29T11:08:26.799 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fbeffd640 1 -- 172.21.15.71:0/688343682 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785236499 0 0) 0x7f1fb802c820 con 0x7f1fb802a640 2024-03-29T11:08:26.799 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fbeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:08:26.799 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:26.799 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:26.799 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fbeffd640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:08:26.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fbeffd640 1 -- 172.21.15.71:0/688343682 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fb802a120 con 0x7f1fb802a640 2024-03-29T11:08:26.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fbeffd640 1 -- 172.21.15.71:0/688343682 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4164604461 0 0) 0x7f1fb0003520 con 0x7f1fb8034850 2024-03-29T11:08:26.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fbeffd640 1 -- 172.21.15.71:0/688343682 >> v1:172.21.15.137:6789/0 conn(0x7f1fb80327d0 legacy=0x7f1fb80cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:26.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fbeffd640 1 -- 172.21.15.71:0/688343682 >> v1:172.21.15.71:6789/0 conn(0x7f1fb802a640 legacy=0x7f1fb80320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:26.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fbeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:08:26.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fbeffd640 1 monclient: found mon.noname-b 2024-03-29T11:08:26.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fbeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:26.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fbeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:08:26.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fbeffd640 1 -- 172.21.15.71:0/688343682 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb8033110 con 0x7f1fb8034850 2024-03-29T11:08:26.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fbeffd640 10 monclient: _finish_auth 0 2024-03-29T11:08:26.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fbeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:08:26.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fbeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:26.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fc50c9640 5 monclient: authenticate success, global_id 4565 2024-03-29T11:08:26.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fc50c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:26.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fbeffd640 1 -- 172.21.15.71:0/688343682 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f1fb4004c10 con 0x7f1fb8034850 2024-03-29T11:08:26.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fbeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:26.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fbeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:08:26.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fbeffd640 10 monclient: dump: 2024-03-29T11:08:26.801 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:26.801 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:26.801 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:26.801 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:26.801 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:26.801 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:26.801 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:26.801 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:26.801 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:26.802 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:26.802 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.819+0000 7f1fc50c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:26.802 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.820+0000 7f1fbeffd640 1 -- 172.21.15.71:0/688343682 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fb4004e30 con 0x7f1fb8034850 2024-03-29T11:08:26.802 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.820+0000 7f1fbeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:26.802 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.820+0000 7f1fc50c9640 10 monclient: get_monmap_and_config success 2024-03-29T11:08:26.802 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.820+0000 7f1fc50c9640 4 set_mon_vals no callback set 2024-03-29T11:08:26.802 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.820+0000 7f1fbeffd640 1 -- 172.21.15.71:0/688343682 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f1fb4001410 con 0x7f1fb8034850 2024-03-29T11:08:26.802 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.820+0000 7f1fbeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:26.802 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.820+0000 7f1fbeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:26.802 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.820+0000 7f1fbeffd640 10 monclient: dump: 2024-03-29T11:08:26.802 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:26.802 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:26.802 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:26.802 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:26.803 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:26.803 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:26.803 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:26.803 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:26.803 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:26.803 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:26.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.821+0000 7f1fc50c9640 10 monclient: shutdown 2024-03-29T11:08:26.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.821+0000 7f1fc50c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:26.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.821+0000 7f1fc50c9640 1 -- 172.21.15.71:0/688343682 >> v1:172.21.15.71:6790/0 conn(0x7f1fb8034850 legacy=0x7f1fb80c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:26.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.822+0000 7f1fc50c9640 1 -- 172.21.15.71:0/688343682 shutdown_connections 2024-03-29T11:08:26.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.822+0000 7f1fc50c9640 1 -- 172.21.15.71:0/688343682 >> 172.21.15.71:0/688343682 conn(0x7f1fb80058f0 msgr2=0x7f1fb8005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:26.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.824+0000 7f1fc50c9640 1 -- 172.21.15.71:0/688343682 shutdown_connections 2024-03-29T11:08:26.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.824+0000 7f1fc50c9640 1 -- 172.21.15.71:0/688343682 wait complete. 2024-03-29T11:08:26.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.825+0000 7f1fc50c9640 10 monclient: build_initial_monmap 2024-03-29T11:08:26.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.825+0000 7f1fc50c9640 10 monclient: monmap: 2024-03-29T11:08:26.804 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:08:26.804 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:08:26.804 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:08:26.804 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:08:26.805 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:26.805 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:26.805 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:08:26.805 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:08:26.805 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:08:26.805 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:26.805 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.826+0000 7f1fc50c9640 1 librados: starting msgr at 2024-03-29T11:08:26.805 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.826+0000 7f1fc50c9640 1 librados: starting objecter 2024-03-29T11:08:26.808 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.826+0000 7f1fc50c9640 1 Processor -- start 2024-03-29T11:08:26.808 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.826+0000 7f1fc50c9640 1 -- start start 2024-03-29T11:08:26.808 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.826+0000 7f1fc50c9640 1 librados: setting wanted keys 2024-03-29T11:08:26.808 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.826+0000 7f1fc50c9640 1 librados: calling monclient init 2024-03-29T11:08:26.808 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.826+0000 7f1fc50c9640 10 monclient: init 2024-03-29T11:08:26.808 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.827+0000 7f1fc50c9640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:26.808 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.827+0000 7f1fc50c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:08:26.808 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.827+0000 7f1fc50c9640 10 monclient(hunting): picked mon.noname-b con 0x7f1fb802a640 addr v1:172.21.15.137:6789/0 2024-03-29T11:08:26.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.827+0000 7f1fc50c9640 10 monclient(hunting): picked mon.noname-a con 0x7f1fb80327d0 addr v1:172.21.15.71:6789/0 2024-03-29T11:08:26.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.827+0000 7f1fc50c9640 10 monclient(hunting): picked mon.noname-c con 0x7f1fb8034850 addr v1:172.21.15.71:6790/0 2024-03-29T11:08:26.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.827+0000 7f1fc50c9640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb80bffa0 con 0x7f1fb80327d0 2024-03-29T11:08:26.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.827+0000 7f1fc50c9640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb80b17b0 con 0x7f1fb802a640 2024-03-29T11:08:26.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.827+0000 7f1fc50c9640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb80c1e80 con 0x7f1fb8034850 2024-03-29T11:08:26.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.827+0000 7f1fc50c9640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:26.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.827+0000 7f1fc50c9640 10 monclient(hunting): authenticate will time out at 1263.085704s 2024-03-29T11:08:26.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.830+0000 7f1fc48c8640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f1fb8034850 0x7f1fb80b10a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:48437/0 (socket says 172.21.15.71:48437) 2024-03-29T11:08:26.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.830+0000 7f1fc48c8640 1 -- 172.21.15.71:0/2281641831 learned_addr learned my addr 172.21.15.71:0/2281641831 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:08:26.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.831+0000 7f1fbcff9640 10 client.?.objecter ms_handle_connect 0x7f1fb8034850 2024-03-29T11:08:26.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.831+0000 7f1fbcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:26.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.831+0000 7f1fbcff9640 1 -- 172.21.15.71:0/2281641831 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771494560 0 0) 0x7f1fb80c1e80 con 0x7f1fb8034850 2024-03-29T11:08:26.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.831+0000 7f1fbcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:08:26.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.831+0000 7f1fbcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:26.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.831+0000 7f1fbcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:26.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.832+0000 7f1fbcff9640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:08:26.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.832+0000 7f1fbcff9640 1 -- 172.21.15.71:0/2281641831 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fa4004eb0 con 0x7f1fb8034850 2024-03-29T11:08:26.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.832+0000 7f1fbcff9640 1 -- 172.21.15.71:0/2281641831 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3870609704 0 0) 0x7f1fa4004eb0 con 0x7f1fb8034850 2024-03-29T11:08:26.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.832+0000 7f1fbcff9640 1 -- 172.21.15.71:0/2281641831 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fb80c1e80 con 0x7f1fb8034850 2024-03-29T11:08:26.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.832+0000 7f1fbcff9640 1 -- 172.21.15.71:0/2281641831 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f1fc004fdd0 con 0x7f1fb8034850 2024-03-29T11:08:26.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.832+0000 7f1fbcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:26.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.832+0000 7f1fbcff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:08:26.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.832+0000 7f1fbcff9640 10 monclient(hunting): dump: 2024-03-29T11:08:26.811 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:26.811 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:26.812 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:26.812 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:26.812 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:26.812 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:26.812 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:26.812 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:26.812 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:26.812 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:26.812 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.832+0000 7f1fbcff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:08:26.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.832+0000 7f1fbcff9640 10 client.?.objecter ms_handle_connect 0x7f1fb80327d0 2024-03-29T11:08:26.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.832+0000 7f1fbcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:26.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.832+0000 7f1fbcff9640 10 client.?.objecter ms_handle_connect 0x7f1fb802a640 2024-03-29T11:08:26.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.832+0000 7f1fbcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:26.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.833+0000 7f1fbcff9640 1 -- 172.21.15.71:0/2281641831 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607338421 0 0) 0x7f1fb80bffa0 con 0x7f1fb80327d0 2024-03-29T11:08:26.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.833+0000 7f1fbcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:08:26.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.833+0000 7f1fbcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:26.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.833+0000 7f1fbcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:26.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.833+0000 7f1fbcff9640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:08:26.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.833+0000 7f1fbcff9640 1 -- 172.21.15.71:0/2281641831 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fa4004eb0 con 0x7f1fb80327d0 2024-03-29T11:08:26.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.833+0000 7f1fbcff9640 1 -- 172.21.15.71:0/2281641831 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3236122221 0 0) 0x7f1fb80c1e80 con 0x7f1fb8034850 2024-03-29T11:08:26.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.833+0000 7f1fbcff9640 1 -- 172.21.15.71:0/2281641831 >> v1:172.21.15.137:6789/0 conn(0x7f1fb802a640 legacy=0x7f1fb816cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:26.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.833+0000 7f1fbcff9640 1 -- 172.21.15.71:0/2281641831 >> v1:172.21.15.71:6789/0 conn(0x7f1fb80327d0 legacy=0x7f1fb80b0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:26.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.833+0000 7f1fbcff9640 10 monclient: _finish_hunting 0 2024-03-29T11:08:26.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.833+0000 7f1fbcff9640 1 monclient: found mon.c 2024-03-29T11:08:26.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.833+0000 7f1fbcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:26.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.833+0000 7f1fbcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:08:26.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.833+0000 7f1fbcff9640 1 -- 172.21.15.71:0/2281641831 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb80befd0 con 0x7f1fb8034850 2024-03-29T11:08:26.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.833+0000 7f1fbcff9640 10 monclient: _finish_auth 0 2024-03-29T11:08:26.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.833+0000 7f1fbcff9640 10 monclient: _check_auth_tickets 2024-03-29T11:08:26.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.833+0000 7f1fbcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:26.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.833+0000 7f1fc50c9640 5 monclient: authenticate success, global_id 4568 2024-03-29T11:08:26.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.833+0000 7f1fc50c9640 10 monclient: _renew_subs 2024-03-29T11:08:26.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.833+0000 7f1fc50c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:08:26.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.833+0000 7f1fc50c9640 1 -- 172.21.15.71:0/2281641831 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fb80c0430 con 0x7f1fb8034850 2024-03-29T11:08:26.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.833+0000 7f1fc50c9640 10 client.4568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:08:26.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.833+0000 7f1fc50c9640 10 monclient: _renew_subs 2024-03-29T11:08:26.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.833+0000 7f1fc50c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:08:26.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.833+0000 7f1fc50c9640 1 -- 172.21.15.71:0/2281641831 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fb816d680 con 0x7f1fb8034850 2024-03-29T11:08:26.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.833+0000 7f1fc50c9640 1 librados: init done 2024-03-29T11:08:26.817 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.835+0000 7f1fbcff9640 1 -- 172.21.15.71:0/2281641831 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fc004f400 con 0x7f1fb8034850 2024-03-29T11:08:26.817 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.835+0000 7f1fbcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:26.818 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.835+0000 7f1fbcff9640 1 -- 172.21.15.71:0/2281641831 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f1fc0075f10 con 0x7f1fb8034850 2024-03-29T11:08:26.818 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.835+0000 7f1fbcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:26.818 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.835+0000 7f1fbcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:26.818 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.835+0000 7f1fbcff9640 10 monclient: dump: 2024-03-29T11:08:26.818 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:26.818 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:26.818 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:26.818 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:26.818 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:26.818 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:26.818 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:26.819 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:26.819 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:26.819 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:26.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.835+0000 7f1fbcff9640 1 -- 172.21.15.71:0/2281641831 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f1fc0067bf0 con 0x7f1fb8034850 2024-03-29T11:08:26.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.836+0000 7f1fbe7fc640 4 set_mon_vals no callback set 2024-03-29T11:08:26.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.837+0000 7f1fbffff640 10 monclient: get_auth_request con 0x7f1fa406dbf0 auth_method 0 2024-03-29T11:08:26.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.837+0000 7f1fc50c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:26.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.837+0000 7f1fbcff9640 10 client.4568.objecter ms_handle_connect 0x7f1fa406dbf0 2024-03-29T11:08:26.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.837+0000 7f1fc50c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:26.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.837+0000 7f1fc50c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:08:26.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.837+0000 7f1fc50c9640 1 -- 172.21.15.71:0/2281641831 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fa0005080 con 0x7f1fb8034850 2024-03-29T11:08:26.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.838+0000 7f1fbcff9640 1 -- 172.21.15.71:0/2281641831 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f1fc00cf590 con 0x7f1fb8034850 2024-03-29T11:08:26.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.838+0000 7f1fbcff9640 10 client.4568.objecter ms_dispatch 0x7f1fc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:08:26.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.838+0000 7f1fbcff9640 3 client.4568.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:08:26.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.838+0000 7f1fbcff9640 3 client.4568.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:08:26.820 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.838+0000 7f1fbcff9640 20 client.4568.objecter dump_active .. 0 homeless 2024-03-29T11:08:26.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.846+0000 7f1fbcff9640 1 -- 172.21.15.71:0/2281641831 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1fc00cb0a0 con 0x7f1fb8034850 2024-03-29T11:08:26.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.846+0000 7f1fbcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:26.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:26.846+0000 7f1fbcff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:08:27.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:27.168+0000 7f1fc50c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:27.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:27.168+0000 7f1fc50c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:27.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:27.168+0000 7f1fc50c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:08:27.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:27.168+0000 7f1fc50c9640 1 -- 172.21.15.71:0/2281641831 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fa0005330 con 0x7f1fb8034850 2024-03-29T11:08:27.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:27.170+0000 7f1fbcff9640 1 -- 172.21.15.71:0/2281641831 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f1fc0067f20 con 0x7f1fb8034850 2024-03-29T11:08:27.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:27.171+0000 7f1fbcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:27.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:27.171+0000 7f1fbcff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:08:27.150 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:08:27.150 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:08:27.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:27.174+0000 7f1fc50c9640 10 librados: watch_flush enter 2024-03-29T11:08:27.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:27.176+0000 7f1fc50c9640 10 librados: watch_flush exit 2024-03-29T11:08:27.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:27.176+0000 7f1fc50c9640 20 client.4568.objecter shutdown clearing up homeless session... 2024-03-29T11:08:27.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:27.176+0000 7f1fc50c9640 10 client.4568.objecter successfully canceled tick 2024-03-29T11:08:27.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:27.176+0000 7f1fc50c9640 1 -- 172.21.15.71:0/2281641831 >> v1:172.21.15.137:6816/973332665 conn(0x7f1fa406dbf0 legacy=0x7f1fa40700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:27.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:27.176+0000 7f1fc50c9640 10 monclient: shutdown 2024-03-29T11:08:27.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:27.176+0000 7f1fc50c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:27.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:27.176+0000 7f1fc50c9640 1 -- 172.21.15.71:0/2281641831 >> v1:172.21.15.71:6790/0 conn(0x7f1fb8034850 legacy=0x7f1fb80b10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:27.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:27.178+0000 7f1fc50c9640 1 -- 172.21.15.71:0/2281641831 shutdown_connections 2024-03-29T11:08:27.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:27.178+0000 7f1fc50c9640 1 -- 172.21.15.71:0/2281641831 >> 172.21.15.71:0/2281641831 conn(0x7f1fb80058f0 msgr2=0x7f1fb80043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:27.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:27.178+0000 7f1fc50c9640 1 -- 172.21.15.71:0/2281641831 shutdown_connections 2024-03-29T11:08:27.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:27.178+0000 7f1fc50c9640 1 -- 172.21.15.71:0/2281641831 wait complete. 2024-03-29T11:08:27.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:27.178+0000 7f1fc50c9640 1 librados: shutdown 2024-03-29T11:08:27.177 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:08:33.984 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:08:34.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.152+0000 7f0e2bfff640 10 monclient: get_monmap_and_config 2024-03-29T11:08:34.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.152+0000 7f0e2bfff640 10 monclient: build_initial_monmap 2024-03-29T11:08:34.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.152+0000 7f0e2bfff640 10 monclient: monmap: 2024-03-29T11:08:34.132 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:08:34.132 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:34.132 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:08:34.152919+0000 2024-03-29T11:08:34.132 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:08:34.152919+0000 2024-03-29T11:08:34.132 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:34.133 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:34.133 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:08:34.133 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:08:34.133 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:08:34.133 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:34.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.153+0000 7f0e2bfff640 1 Processor -- start 2024-03-29T11:08:34.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.154+0000 7f0e2bfff640 1 -- start start 2024-03-29T11:08:34.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.154+0000 7f0e2bfff640 10 monclient: init 2024-03-29T11:08:34.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.154+0000 7f0e2bfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:34.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.154+0000 7f0e2bfff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:08:34.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.154+0000 7f0e2bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f0e24006eb0 addr v1:172.21.15.137:6789/0 2024-03-29T11:08:34.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.154+0000 7f0e2bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f0e24038f10 addr v1:172.21.15.71:6790/0 2024-03-29T11:08:34.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.154+0000 7f0e2bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f0e24036690 addr v1:172.21.15.71:6789/0 2024-03-29T11:08:34.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.154+0000 7f0e2bfff640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e24033c90 con 0x7f0e24036690 2024-03-29T11:08:34.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.154+0000 7f0e2bfff640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e240389f0 con 0x7f0e24006eb0 2024-03-29T11:08:34.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.154+0000 7f0e2bfff640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e24030890 con 0x7f0e24038f10 2024-03-29T11:08:34.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.154+0000 7f0e2bfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:34.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.154+0000 7f0e2bfff640 10 monclient(hunting): authenticate will time out at 1270.413057s 2024-03-29T11:08:34.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.156+0000 7f0e2a7fc640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f0e24038f10 0x7f0e24039330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:49554/0 (socket says 172.21.15.71:49554) 2024-03-29T11:08:34.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.156+0000 7f0e2a7fc640 1 -- 172.21.15.71:0/1656017612 learned_addr learned my addr 172.21.15.71:0/1656017612 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:08:34.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.158+0000 7f0e29ffb640 1 -- 172.21.15.71:0/1656017612 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210950678 0 0) 0x7f0e24030890 con 0x7f0e24038f10 2024-03-29T11:08:34.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.158+0000 7f0e29ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:08:34.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.158+0000 7f0e29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:34.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.158+0000 7f0e29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:34.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.158+0000 7f0e29ffb640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:08:34.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.158+0000 7f0e29ffb640 1 -- 172.21.15.71:0/1656017612 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e14003520 con 0x7f0e24038f10 2024-03-29T11:08:34.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.158+0000 7f0e29ffb640 1 -- 172.21.15.71:0/1656017612 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3661325510 0 0) 0x7f0e14003520 con 0x7f0e24038f10 2024-03-29T11:08:34.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.158+0000 7f0e29ffb640 1 -- 172.21.15.71:0/1656017612 >> v1:172.21.15.137:6789/0 conn(0x7f0e24006eb0 legacy=0x7f0e24031760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:08:34.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.160+0000 7f0e29ffb640 1 -- 172.21.15.71:0/1656017612 >> v1:172.21.15.71:6789/0 conn(0x7f0e24036690 legacy=0x7f0e24036b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:34.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.160+0000 7f0e29ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:08:34.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.160+0000 7f0e29ffb640 1 monclient: found mon.noname-b 2024-03-29T11:08:34.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.160+0000 7f0e29ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:34.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.160+0000 7f0e29ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:08:34.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.160+0000 7f0e29ffb640 1 -- 172.21.15.71:0/1656017612 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e240cbc40 con 0x7f0e24038f10 2024-03-29T11:08:34.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.160+0000 7f0e29ffb640 10 monclient: _finish_auth 0 2024-03-29T11:08:34.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.160+0000 7f0e29ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:08:34.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.160+0000 7f0e29ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:34.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.160+0000 7f0e29ffb640 1 -- 172.21.15.71:0/1656017612 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f0e2c04f730 con 0x7f0e24038f10 2024-03-29T11:08:34.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.160+0000 7f0e29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:34.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.160+0000 7f0e29ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:08:34.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.160+0000 7f0e29ffb640 10 monclient: dump: 2024-03-29T11:08:34.139 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:34.139 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:34.139 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:34.139 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:34.140 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:34.140 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:34.140 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:34.140 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:34.140 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:34.140 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:34.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.160+0000 7f0e2bfff640 5 monclient: authenticate success, global_id 4571 2024-03-29T11:08:34.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.160+0000 7f0e2bfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:34.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.161+0000 7f0e29ffb640 1 -- 172.21.15.71:0/1656017612 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e2c075f10 con 0x7f0e24038f10 2024-03-29T11:08:34.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.161+0000 7f0e29ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:34.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.161+0000 7f0e2bfff640 10 monclient: get_monmap_and_config success 2024-03-29T11:08:34.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.161+0000 7f0e2bfff640 4 set_mon_vals no callback set 2024-03-29T11:08:34.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.162+0000 7f0e29ffb640 1 -- 172.21.15.71:0/1656017612 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f0e2c0758b0 con 0x7f0e24038f10 2024-03-29T11:08:34.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.162+0000 7f0e29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:34.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.162+0000 7f0e29ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:34.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.162+0000 7f0e29ffb640 10 monclient: dump: 2024-03-29T11:08:34.141 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:34.141 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:34.141 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:34.142 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:34.142 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:34.142 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:34.142 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:34.142 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:34.142 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:34.142 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:34.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.162+0000 7f0e2bfff640 10 monclient: shutdown 2024-03-29T11:08:34.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.162+0000 7f0e2bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:34.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.162+0000 7f0e2bfff640 1 -- 172.21.15.71:0/1656017612 >> v1:172.21.15.71:6790/0 conn(0x7f0e24038f10 legacy=0x7f0e24039330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:34.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.163+0000 7f0e2bfff640 1 -- 172.21.15.71:0/1656017612 shutdown_connections 2024-03-29T11:08:34.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.163+0000 7f0e2bfff640 1 -- 172.21.15.71:0/1656017612 >> 172.21.15.71:0/1656017612 conn(0x7f0e24005c20 msgr2=0x7f0e24006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:34.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.163+0000 7f0e2bfff640 1 -- 172.21.15.71:0/1656017612 shutdown_connections 2024-03-29T11:08:34.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.163+0000 7f0e2bfff640 1 -- 172.21.15.71:0/1656017612 wait complete. 2024-03-29T11:08:34.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.165+0000 7f0e2bfff640 10 monclient: build_initial_monmap 2024-03-29T11:08:34.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.165+0000 7f0e2bfff640 10 monclient: monmap: 2024-03-29T11:08:34.144 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:08:34.144 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:08:34.144 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:08:34.144 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:08:34.144 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:34.144 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:34.145 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:08:34.145 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:08:34.145 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:08:34.145 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:34.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.165+0000 7f0e2bfff640 1 librados: starting msgr at 2024-03-29T11:08:34.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.165+0000 7f0e2bfff640 1 librados: starting objecter 2024-03-29T11:08:34.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.165+0000 7f0e2bfff640 1 Processor -- start 2024-03-29T11:08:34.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.166+0000 7f0e2bfff640 1 -- start start 2024-03-29T11:08:34.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.166+0000 7f0e2bfff640 1 librados: setting wanted keys 2024-03-29T11:08:34.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.166+0000 7f0e2bfff640 1 librados: calling monclient init 2024-03-29T11:08:34.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.166+0000 7f0e2bfff640 10 monclient: init 2024-03-29T11:08:34.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.166+0000 7f0e2bfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:34.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.166+0000 7f0e2bfff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:08:34.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.166+0000 7f0e2bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f0e24006eb0 addr v1:172.21.15.71:6790/0 2024-03-29T11:08:34.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.166+0000 7f0e2bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f0e24036690 addr v1:172.21.15.137:6789/0 2024-03-29T11:08:34.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.166+0000 7f0e2bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f0e2400d6a0 addr v1:172.21.15.71:6789/0 2024-03-29T11:08:34.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.166+0000 7f0e2bfff640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e2400add0 con 0x7f0e2400d6a0 2024-03-29T11:08:34.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.166+0000 7f0e2bfff640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e2400bc50 con 0x7f0e24036690 2024-03-29T11:08:34.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.166+0000 7f0e2bfff640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e2400bf30 con 0x7f0e24006eb0 2024-03-29T11:08:34.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.166+0000 7f0e2bfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:34.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.166+0000 7f0e2bfff640 10 monclient(hunting): authenticate will time out at 1270.425039s 2024-03-29T11:08:34.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.167+0000 7f0e2affd640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f0e24006eb0 0x7f0e2400a470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:50964/0 (socket says 172.21.15.71:50964) 2024-03-29T11:08:34.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.167+0000 7f0e2affd640 1 -- 172.21.15.71:0/3789661996 learned_addr learned my addr 172.21.15.71:0/3789661996 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:08:34.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.167+0000 7f0e13fff640 10 client.?.objecter ms_handle_connect 0x7f0e24006eb0 2024-03-29T11:08:34.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.167+0000 7f0e13fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:34.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.168+0000 7f0e13fff640 1 -- 172.21.15.71:0/3789661996 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2990277925 0 0) 0x7f0e2400bf30 con 0x7f0e24006eb0 2024-03-29T11:08:34.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.168+0000 7f0e13fff640 10 monclient(hunting): _negotiate 2024-03-29T11:08:34.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.168+0000 7f0e13fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:34.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.168+0000 7f0e13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:34.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.168+0000 7f0e13fff640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:08:34.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.168+0000 7f0e13fff640 1 -- 172.21.15.71:0/3789661996 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e18004eb0 con 0x7f0e24006eb0 2024-03-29T11:08:34.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.171+0000 7f0e13fff640 1 -- 172.21.15.71:0/3789661996 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3230877912 0 0) 0x7f0e18004eb0 con 0x7f0e24006eb0 2024-03-29T11:08:34.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.171+0000 7f0e13fff640 1 -- 172.21.15.71:0/3789661996 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e18005300 con 0x7f0e24006eb0 2024-03-29T11:08:34.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.171+0000 7f0e13fff640 1 -- 172.21.15.71:0/3789661996 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f0e20001830 con 0x7f0e24006eb0 2024-03-29T11:08:34.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.171+0000 7f0e13fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:34.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.171+0000 7f0e13fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:08:34.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.171+0000 7f0e13fff640 10 monclient(hunting): dump: 2024-03-29T11:08:34.150 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:34.150 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:34.150 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:34.151 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:34.151 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:34.151 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:34.151 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:34.151 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:34.151 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:34.151 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:34.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.171+0000 7f0e13fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:08:34.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.172+0000 7f0e13fff640 10 client.?.objecter ms_handle_connect 0x7f0e2400d6a0 2024-03-29T11:08:34.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.172+0000 7f0e13fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:34.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.172+0000 7f0e13fff640 1 -- 172.21.15.71:0/3789661996 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588128196 0 0) 0x7f0e2400add0 con 0x7f0e2400d6a0 2024-03-29T11:08:34.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.172+0000 7f0e13fff640 10 monclient(hunting): _negotiate 2024-03-29T11:08:34.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.172+0000 7f0e13fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:34.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.172+0000 7f0e13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:34.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.172+0000 7f0e13fff640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:08:34.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.172+0000 7f0e13fff640 1 -- 172.21.15.71:0/3789661996 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e180058a0 con 0x7f0e2400d6a0 2024-03-29T11:08:34.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.172+0000 7f0e13fff640 1 -- 172.21.15.71:0/3789661996 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1209602136 0 0) 0x7f0e18005300 con 0x7f0e24006eb0 2024-03-29T11:08:34.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.172+0000 7f0e13fff640 1 -- 172.21.15.71:0/3789661996 >> v1:172.21.15.137:6789/0 conn(0x7f0e24036690 legacy=0x7f0e2402dbe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:08:34.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.173+0000 7f0e13fff640 1 -- 172.21.15.71:0/3789661996 >> v1:172.21.15.71:6789/0 conn(0x7f0e2400d6a0 legacy=0x7f0e2416bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:34.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.173+0000 7f0e13fff640 10 monclient: _finish_hunting 0 2024-03-29T11:08:34.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.173+0000 7f0e13fff640 1 monclient: found mon.c 2024-03-29T11:08:34.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.173+0000 7f0e13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:34.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.173+0000 7f0e13fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:08:34.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.173+0000 7f0e13fff640 1 -- 172.21.15.71:0/3789661996 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e2402e2f0 con 0x7f0e24006eb0 2024-03-29T11:08:34.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.173+0000 7f0e13fff640 10 monclient: _finish_auth 0 2024-03-29T11:08:34.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.173+0000 7f0e13fff640 10 monclient: _check_auth_tickets 2024-03-29T11:08:34.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.173+0000 7f0e13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:34.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.174+0000 7f0e2bfff640 5 monclient: authenticate success, global_id 4574 2024-03-29T11:08:34.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.174+0000 7f0e2bfff640 10 monclient: _renew_subs 2024-03-29T11:08:34.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.174+0000 7f0e2bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:08:34.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.174+0000 7f0e2bfff640 1 -- 172.21.15.71:0/3789661996 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e240098d0 con 0x7f0e24006eb0 2024-03-29T11:08:34.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.174+0000 7f0e2bfff640 10 client.4574.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:08:34.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.174+0000 7f0e2bfff640 10 monclient: _renew_subs 2024-03-29T11:08:34.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.174+0000 7f0e2bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:08:34.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.174+0000 7f0e2bfff640 1 -- 172.21.15.71:0/3789661996 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e2416d6a0 con 0x7f0e24006eb0 2024-03-29T11:08:34.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.174+0000 7f0e2bfff640 1 librados: init done 2024-03-29T11:08:34.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.175+0000 7f0e13fff640 1 -- 172.21.15.71:0/3789661996 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e20004900 con 0x7f0e24006eb0 2024-03-29T11:08:34.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.175+0000 7f0e13fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:34.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.175+0000 7f0e13fff640 1 -- 172.21.15.71:0/3789661996 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f0e20004c10 con 0x7f0e24006eb0 2024-03-29T11:08:34.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.175+0000 7f0e13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:34.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.175+0000 7f0e13fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:34.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.175+0000 7f0e13fff640 10 monclient: dump: 2024-03-29T11:08:34.155 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:34.155 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:34.155 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:34.155 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:34.155 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:34.155 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:34.155 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:34.155 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:34.156 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:34.156 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:34.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.175+0000 7f0e297fa640 4 set_mon_vals no callback set 2024-03-29T11:08:34.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.176+0000 7f0e2bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:34.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.176+0000 7f0e2bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:34.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.176+0000 7f0e2bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:08:34.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.176+0000 7f0e2bfff640 1 -- 172.21.15.71:0/3789661996 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e0c005080 con 0x7f0e24006eb0 2024-03-29T11:08:34.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.180+0000 7f0e13fff640 1 -- 172.21.15.71:0/3789661996 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f0e2001abe0 con 0x7f0e24006eb0 2024-03-29T11:08:34.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.180+0000 7f0e2a7fc640 10 monclient: get_auth_request con 0x7f0e1806df80 auth_method 0 2024-03-29T11:08:34.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.180+0000 7f0e13fff640 1 -- 172.21.15.71:0/3789661996 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f0e20054890 con 0x7f0e24006eb0 2024-03-29T11:08:34.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.180+0000 7f0e13fff640 10 client.4574.objecter ms_dispatch 0x7f0e2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:08:34.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.180+0000 7f0e13fff640 3 client.4574.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:08:34.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.180+0000 7f0e13fff640 3 client.4574.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:08:34.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.180+0000 7f0e13fff640 20 client.4574.objecter dump_active .. 0 homeless 2024-03-29T11:08:34.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.181+0000 7f0e13fff640 10 client.4574.objecter ms_handle_connect 0x7f0e1806df80 2024-03-29T11:08:34.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.191+0000 7f0e13fff640 1 -- 172.21.15.71:0/3789661996 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0e20050b30 con 0x7f0e24006eb0 2024-03-29T11:08:34.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.191+0000 7f0e13fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:34.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.191+0000 7f0e13fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:08:34.501 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.522+0000 7f0e2bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:34.502 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.522+0000 7f0e2bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:34.502 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.522+0000 7f0e2bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:08:34.502 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.522+0000 7f0e2bfff640 1 -- 172.21.15.71:0/3789661996 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e0c005330 con 0x7f0e24006eb0 2024-03-29T11:08:34.505 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.527+0000 7f0e13fff640 1 -- 172.21.15.71:0/3789661996 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f0e20050b30 con 0x7f0e24006eb0 2024-03-29T11:08:34.505 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.527+0000 7f0e13fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:34.505 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.527+0000 7f0e13fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:08:34.506 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:08:34.506 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:08:34.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.530+0000 7f0e2bfff640 10 librados: watch_flush enter 2024-03-29T11:08:34.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.530+0000 7f0e2bfff640 10 librados: watch_flush exit 2024-03-29T11:08:34.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.530+0000 7f0e2bfff640 20 client.4574.objecter shutdown clearing up homeless session... 2024-03-29T11:08:34.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.530+0000 7f0e2bfff640 10 client.4574.objecter successfully canceled tick 2024-03-29T11:08:34.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.531+0000 7f0e2bfff640 1 -- 172.21.15.71:0/3789661996 >> v1:172.21.15.137:6816/973332665 conn(0x7f0e1806df80 legacy=0x7f0e18070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:34.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.531+0000 7f0e2bfff640 10 monclient: shutdown 2024-03-29T11:08:34.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.531+0000 7f0e2bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:34.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.531+0000 7f0e2bfff640 1 -- 172.21.15.71:0/3789661996 >> v1:172.21.15.71:6790/0 conn(0x7f0e24006eb0 legacy=0x7f0e2400a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:34.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.531+0000 7f0e2bfff640 1 -- 172.21.15.71:0/3789661996 shutdown_connections 2024-03-29T11:08:34.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.531+0000 7f0e2bfff640 1 -- 172.21.15.71:0/3789661996 >> 172.21.15.71:0/3789661996 conn(0x7f0e24005c20 msgr2=0x7f0e2403b340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:34.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.531+0000 7f0e2bfff640 1 -- 172.21.15.71:0/3789661996 shutdown_connections 2024-03-29T11:08:34.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.532+0000 7f0e2bfff640 1 -- 172.21.15.71:0/3789661996 wait complete. 2024-03-29T11:08:34.512 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:34.534+0000 7f0e2bfff640 1 librados: shutdown 2024-03-29T11:08:34.529 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:08:41.336 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:08:41.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.505+0000 7fe28bfff640 10 monclient: get_monmap_and_config 2024-03-29T11:08:41.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.505+0000 7fe28bfff640 10 monclient: build_initial_monmap 2024-03-29T11:08:41.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.505+0000 7fe28bfff640 10 monclient: monmap: 2024-03-29T11:08:41.484 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:08:41.484 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:41.484 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:08:41.506505+0000 2024-03-29T11:08:41.484 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:08:41.506505+0000 2024-03-29T11:08:41.484 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:41.484 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:41.484 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:08:41.484 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:08:41.485 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:08:41.485 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:41.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.509+0000 7fe28bfff640 1 Processor -- start 2024-03-29T11:08:41.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.509+0000 7fe28bfff640 1 -- start start 2024-03-29T11:08:41.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.509+0000 7fe28bfff640 10 monclient: init 2024-03-29T11:08:41.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.510+0000 7fe28bfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:41.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.510+0000 7fe28bfff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:08:41.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.510+0000 7fe28bfff640 10 monclient(hunting): picked mon.noname-a con 0x7fe28c14a400 addr v1:172.21.15.71:6789/0 2024-03-29T11:08:41.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.510+0000 7fe28bfff640 10 monclient(hunting): picked mon.noname-b con 0x7fe28c0aa500 addr v1:172.21.15.71:6790/0 2024-03-29T11:08:41.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.510+0000 7fe28bfff640 10 monclient(hunting): picked mon.noname-c con 0x7fe28c149800 addr v1:172.21.15.137:6789/0 2024-03-29T11:08:41.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.510+0000 7fe28bfff640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe28c0aca50 con 0x7fe28c14a400 2024-03-29T11:08:41.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.510+0000 7fe28bfff640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe28c149cb0 con 0x7fe28c149800 2024-03-29T11:08:41.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.510+0000 7fe28bfff640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe28c0acc80 con 0x7fe28c0aa500 2024-03-29T11:08:41.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.510+0000 7fe28bfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:41.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.510+0000 7fe28bfff640 10 monclient(hunting): authenticate will time out at 1277.768742s 2024-03-29T11:08:41.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.512+0000 7fe28a7fc640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fe28c0aa500 0x7fe28c0ad160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:42730/0 (socket says 172.21.15.71:42730) 2024-03-29T11:08:41.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.512+0000 7fe28a7fc640 1 -- 172.21.15.71:0/4242397078 learned_addr learned my addr 172.21.15.71:0/4242397078 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:08:41.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.513+0000 7fe289ffb640 1 -- 172.21.15.71:0/4242397078 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3246221633 0 0) 0x7fe28c0acc80 con 0x7fe28c0aa500 2024-03-29T11:08:41.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.513+0000 7fe289ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:08:41.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.513+0000 7fe289ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:41.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.513+0000 7fe289ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:41.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.513+0000 7fe289ffb640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:08:41.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.513+0000 7fe289ffb640 1 -- 172.21.15.71:0/4242397078 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe274003520 con 0x7fe28c0aa500 2024-03-29T11:08:41.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.513+0000 7fe289ffb640 1 -- 172.21.15.71:0/4242397078 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2004840271 0 0) 0x7fe274003520 con 0x7fe28c0aa500 2024-03-29T11:08:41.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.513+0000 7fe289ffb640 1 -- 172.21.15.71:0/4242397078 >> v1:172.21.15.137:6789/0 conn(0x7fe28c149800 legacy=0x7fe28c151ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:08:41.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.516+0000 7fe289ffb640 1 -- 172.21.15.71:0/4242397078 >> v1:172.21.15.71:6789/0 conn(0x7fe28c14a400 legacy=0x7fe28c0aac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:41.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.516+0000 7fe289ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:08:41.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.516+0000 7fe289ffb640 1 monclient: found mon.noname-b 2024-03-29T11:08:41.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.516+0000 7fe289ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:41.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.516+0000 7fe289ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:08:41.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.516+0000 7fe289ffb640 1 -- 172.21.15.71:0/4242397078 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28c0ac350 con 0x7fe28c0aa500 2024-03-29T11:08:41.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.516+0000 7fe289ffb640 10 monclient: _finish_auth 0 2024-03-29T11:08:41.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.516+0000 7fe289ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:08:41.498 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.516+0000 7fe289ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:41.498 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.516+0000 7fe289ffb640 1 -- 172.21.15.71:0/4242397078 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fe280001690 con 0x7fe28c0aa500 2024-03-29T11:08:41.498 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.516+0000 7fe28bfff640 5 monclient: authenticate success, global_id 4577 2024-03-29T11:08:41.498 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.516+0000 7fe28bfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:41.498 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.516+0000 7fe289ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:41.498 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.516+0000 7fe289ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:08:41.498 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.516+0000 7fe289ffb640 10 monclient: dump: 2024-03-29T11:08:41.498 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:41.498 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:41.498 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:41.498 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:41.498 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:41.498 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:41.499 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:41.499 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:41.499 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:41.499 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:41.499 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.516+0000 7fe28bfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:41.499 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.516+0000 7fe289ffb640 1 -- 172.21.15.71:0/4242397078 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe280001860 con 0x7fe28c0aa500 2024-03-29T11:08:41.499 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.516+0000 7fe289ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:41.499 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.516+0000 7fe28bfff640 10 monclient: get_monmap_and_config success 2024-03-29T11:08:41.499 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.516+0000 7fe289ffb640 1 -- 172.21.15.71:0/4242397078 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fe280001cb0 con 0x7fe28c0aa500 2024-03-29T11:08:41.499 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.516+0000 7fe28bfff640 4 set_mon_vals no callback set 2024-03-29T11:08:41.499 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.516+0000 7fe289ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:41.499 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.516+0000 7fe289ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:41.499 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.516+0000 7fe289ffb640 10 monclient: dump: 2024-03-29T11:08:41.499 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:41.500 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:41.500 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:41.500 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:41.500 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:41.500 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:41.500 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:41.500 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:41.500 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:41.500 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:41.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.517+0000 7fe28bfff640 10 monclient: shutdown 2024-03-29T11:08:41.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.517+0000 7fe28bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:41.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.517+0000 7fe28bfff640 1 -- 172.21.15.71:0/4242397078 >> v1:172.21.15.71:6790/0 conn(0x7fe28c0aa500 legacy=0x7fe28c0ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:41.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.517+0000 7fe28bfff640 1 -- 172.21.15.71:0/4242397078 shutdown_connections 2024-03-29T11:08:41.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.517+0000 7fe28bfff640 1 -- 172.21.15.71:0/4242397078 >> 172.21.15.71:0/4242397078 conn(0x7fe28c0b3a30 msgr2=0x7fe28c0b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:41.500 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.518+0000 7fe28bfff640 1 -- 172.21.15.71:0/4242397078 shutdown_connections 2024-03-29T11:08:41.502 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.519+0000 7fe28bfff640 1 -- 172.21.15.71:0/4242397078 wait complete. 2024-03-29T11:08:41.502 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.521+0000 7fe28bfff640 10 monclient: build_initial_monmap 2024-03-29T11:08:41.502 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.521+0000 7fe28bfff640 10 monclient: monmap: 2024-03-29T11:08:41.502 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:08:41.502 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:08:41.502 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:08:41.502 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:08:41.502 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:41.502 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:41.502 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:08:41.502 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:08:41.503 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:08:41.503 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:41.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.521+0000 7fe28bfff640 1 librados: starting msgr at 2024-03-29T11:08:41.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.521+0000 7fe28bfff640 1 librados: starting objecter 2024-03-29T11:08:41.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.522+0000 7fe28bfff640 1 Processor -- start 2024-03-29T11:08:41.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.522+0000 7fe28bfff640 1 -- start start 2024-03-29T11:08:41.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.522+0000 7fe28bfff640 1 librados: setting wanted keys 2024-03-29T11:08:41.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.522+0000 7fe28bfff640 1 librados: calling monclient init 2024-03-29T11:08:41.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.522+0000 7fe28bfff640 10 monclient: init 2024-03-29T11:08:41.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.523+0000 7fe28bfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:41.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.523+0000 7fe28bfff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:08:41.503 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.523+0000 7fe28bfff640 10 monclient(hunting): picked mon.noname-a con 0x7fe28c149800 addr v1:172.21.15.71:6789/0 2024-03-29T11:08:41.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.523+0000 7fe28bfff640 10 monclient(hunting): picked mon.noname-b con 0x7fe28c14a400 addr v1:172.21.15.137:6789/0 2024-03-29T11:08:41.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.526+0000 7fe28bfff640 10 monclient(hunting): picked mon.noname-c con 0x7fe28c144fc0 addr v1:172.21.15.71:6790/0 2024-03-29T11:08:41.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.526+0000 7fe28a7fc640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fe28c14a400 0x7fe28c1db980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:33889/0 (socket says 172.21.15.71:33889) 2024-03-29T11:08:41.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.526+0000 7fe28a7fc640 1 -- 172.21.15.71:0/4201798576 learned_addr learned my addr 172.21.15.71:0/4201798576 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:08:41.508 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.526+0000 7fe28bfff640 1 -- 172.21.15.71:0/4201798576 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe28c147040 con 0x7fe28c149800 2024-03-29T11:08:41.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.526+0000 7fe28bfff640 1 -- 172.21.15.71:0/4201798576 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe28c1db4a0 con 0x7fe28c14a400 2024-03-29T11:08:41.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.526+0000 7fe28bfff640 1 -- 172.21.15.71:0/4201798576 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe28c1463e0 con 0x7fe28c144fc0 2024-03-29T11:08:41.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.526+0000 7fe28bfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:41.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.526+0000 7fe28bfff640 10 monclient(hunting): authenticate will time out at 1277.784939s 2024-03-29T11:08:41.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.527+0000 7fe2737fe640 10 client.?.objecter ms_handle_connect 0x7fe28c14a400 2024-03-29T11:08:41.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.527+0000 7fe2737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:41.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.527+0000 7fe2737fe640 10 client.?.objecter ms_handle_connect 0x7fe28c144fc0 2024-03-29T11:08:41.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.527+0000 7fe2737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:41.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.527+0000 7fe2737fe640 1 -- 172.21.15.71:0/4201798576 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223280205 0 0) 0x7fe28c1db4a0 con 0x7fe28c14a400 2024-03-29T11:08:41.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.527+0000 7fe2737fe640 10 monclient(hunting): _negotiate 2024-03-29T11:08:41.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.527+0000 7fe2737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:41.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.527+0000 7fe2737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:41.509 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.527+0000 7fe2737fe640 10 monclient(hunting): my global_id is 4477 2024-03-29T11:08:41.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.527+0000 7fe2737fe640 1 -- 172.21.15.71:0/4201798576 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe278004eb0 con 0x7fe28c14a400 2024-03-29T11:08:41.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.527+0000 7fe2737fe640 1 -- 172.21.15.71:0/4201798576 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1708351228 0 0) 0x7fe28c1463e0 con 0x7fe28c144fc0 2024-03-29T11:08:41.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.527+0000 7fe2737fe640 10 monclient(hunting): _negotiate 2024-03-29T11:08:41.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.527+0000 7fe2737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:41.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.527+0000 7fe2737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:41.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.527+0000 7fe2737fe640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:08:41.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.527+0000 7fe2737fe640 1 -- 172.21.15.71:0/4201798576 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe28c1db4a0 con 0x7fe28c144fc0 2024-03-29T11:08:41.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.528+0000 7fe2737fe640 1 -- 172.21.15.71:0/4201798576 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3310826507 0 0) 0x7fe28c1db4a0 con 0x7fe28c144fc0 2024-03-29T11:08:41.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.528+0000 7fe2737fe640 1 -- 172.21.15.71:0/4201798576 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe28c1463e0 con 0x7fe28c144fc0 2024-03-29T11:08:41.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.528+0000 7fe2737fe640 1 -- 172.21.15.71:0/4201798576 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1763969261 0 0) 0x7fe278004eb0 con 0x7fe28c14a400 2024-03-29T11:08:41.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.528+0000 7fe2737fe640 1 -- 172.21.15.71:0/4201798576 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe28c1db4a0 con 0x7fe28c14a400 2024-03-29T11:08:41.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.528+0000 7fe2737fe640 1 -- 172.21.15.71:0/4201798576 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fe284001720 con 0x7fe28c144fc0 2024-03-29T11:08:41.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.528+0000 7fe2737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:41.510 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.528+0000 7fe2737fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:08:41.511 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.528+0000 7fe2737fe640 10 monclient(hunting): dump: 2024-03-29T11:08:41.511 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:41.511 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:41.511 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:41.511 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:41.511 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:41.511 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:41.511 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:41.511 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:41.511 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:41.511 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:41.511 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.528+0000 7fe2737fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:08:41.511 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.528+0000 7fe2737fe640 10 client.?.objecter ms_handle_connect 0x7fe28c149800 2024-03-29T11:08:41.511 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.528+0000 7fe2737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:41.512 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.528+0000 7fe2737fe640 1 -- 172.21.15.71:0/4201798576 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fe280001b70 con 0x7fe28c14a400 2024-03-29T11:08:41.512 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.528+0000 7fe2737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:41.512 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.528+0000 7fe2737fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:08:41.512 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.528+0000 7fe2737fe640 10 monclient(hunting): dump: 2024-03-29T11:08:41.512 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:41.512 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:41.512 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:41.512 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:41.512 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:41.512 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:41.512 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:41.512 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:41.512 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:41.512 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:41.512 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.528+0000 7fe2737fe640 1 -- 172.21.15.71:0/4201798576 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2940371121 0 0) 0x7fe28c1463e0 con 0x7fe28c144fc0 2024-03-29T11:08:41.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.528+0000 7fe2737fe640 1 -- 172.21.15.71:0/4201798576 >> v1:172.21.15.137:6789/0 conn(0x7fe28c14a400 legacy=0x7fe28c1db980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:41.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.528+0000 7fe2737fe640 1 -- 172.21.15.71:0/4201798576 >> v1:172.21.15.71:6789/0 conn(0x7fe28c149800 legacy=0x7fe28c145cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:41.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.528+0000 7fe2737fe640 10 monclient: _finish_hunting 0 2024-03-29T11:08:41.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.529+0000 7fe2737fe640 1 monclient: found mon.c 2024-03-29T11:08:41.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.529+0000 7fe2737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:41.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.529+0000 7fe2737fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:08:41.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.529+0000 7fe2737fe640 1 -- 172.21.15.71:0/4201798576 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28c146570 con 0x7fe28c144fc0 2024-03-29T11:08:41.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.529+0000 7fe2737fe640 10 monclient: _finish_auth 0 2024-03-29T11:08:41.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.529+0000 7fe2737fe640 10 monclient: _check_auth_tickets 2024-03-29T11:08:41.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.529+0000 7fe2737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:41.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.529+0000 7fe28bfff640 5 monclient: authenticate success, global_id 4580 2024-03-29T11:08:41.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.529+0000 7fe28bfff640 10 monclient: _renew_subs 2024-03-29T11:08:41.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.529+0000 7fe28bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:08:41.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.529+0000 7fe28bfff640 1 -- 172.21.15.71:0/4201798576 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe28c138870 con 0x7fe28c144fc0 2024-03-29T11:08:41.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.529+0000 7fe28bfff640 10 client.4580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:08:41.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.529+0000 7fe28bfff640 10 monclient: _renew_subs 2024-03-29T11:08:41.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.529+0000 7fe28bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:08:41.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.529+0000 7fe28bfff640 1 -- 172.21.15.71:0/4201798576 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe28c1dc270 con 0x7fe28c144fc0 2024-03-29T11:08:41.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.530+0000 7fe28bfff640 1 librados: init done 2024-03-29T11:08:41.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.530+0000 7fe2737fe640 1 -- 172.21.15.71:0/4201798576 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe284001880 con 0x7fe28c144fc0 2024-03-29T11:08:41.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.530+0000 7fe2737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:41.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.530+0000 7fe2737fe640 1 -- 172.21.15.71:0/4201798576 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fe2840035f0 con 0x7fe28c144fc0 2024-03-29T11:08:41.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.530+0000 7fe2737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:41.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.530+0000 7fe2737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:41.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.531+0000 7fe2737fe640 10 monclient: dump: 2024-03-29T11:08:41.516 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:41.516 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:41.516 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:41.517 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:41.517 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:41.517 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:41.517 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:41.517 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:41.517 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:41.517 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:41.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.532+0000 7fe2897fa640 4 set_mon_vals no callback set 2024-03-29T11:08:41.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.533+0000 7fe2737fe640 1 -- 172.21.15.71:0/4201798576 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7fe2840195c0 con 0x7fe28c144fc0 2024-03-29T11:08:41.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.535+0000 7fe28bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:41.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.535+0000 7fe28bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:41.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.535+0000 7fe28bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:08:41.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.535+0000 7fe28bfff640 1 -- 172.21.15.71:0/4201798576 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe26c005080 con 0x7fe28c144fc0 2024-03-29T11:08:41.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.535+0000 7fe2737fe640 1 -- 172.21.15.71:0/4201798576 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7fe284083570 con 0x7fe28c144fc0 2024-03-29T11:08:41.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.535+0000 7fe2737fe640 10 client.4580.objecter ms_dispatch 0x7fe28c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:08:41.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.535+0000 7fe2737fe640 3 client.4580.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:08:41.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.535+0000 7fe2737fe640 3 client.4580.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:08:41.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.535+0000 7fe2737fe640 20 client.4580.objecter dump_active .. 0 homeless 2024-03-29T11:08:41.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.536+0000 7fe28affd640 10 monclient: get_auth_request con 0x7fe27806deb0 auth_method 0 2024-03-29T11:08:41.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.539+0000 7fe2737fe640 10 client.4580.objecter ms_handle_connect 0x7fe27806deb0 2024-03-29T11:08:41.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.545+0000 7fe2737fe640 1 -- 172.21.15.71:0/4201798576 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe28404ec80 con 0x7fe28c144fc0 2024-03-29T11:08:41.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.545+0000 7fe2737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:41.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.545+0000 7fe2737fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:08:41.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.861+0000 7fe28bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:41.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.861+0000 7fe28bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:41.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.861+0000 7fe28bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:08:41.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.861+0000 7fe28bfff640 1 -- 172.21.15.71:0/4201798576 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe26c005330 con 0x7fe28c144fc0 2024-03-29T11:08:41.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.863+0000 7fe2737fe640 1 -- 172.21.15.71:0/4201798576 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7fe284050d80 con 0x7fe28c144fc0 2024-03-29T11:08:41.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.865+0000 7fe2737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:41.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.865+0000 7fe2737fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:08:41.843 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:08:41.844 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:08:41.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.868+0000 7fe28bfff640 10 librados: watch_flush enter 2024-03-29T11:08:41.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.868+0000 7fe28bfff640 10 librados: watch_flush exit 2024-03-29T11:08:41.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.868+0000 7fe28bfff640 20 client.4580.objecter shutdown clearing up homeless session... 2024-03-29T11:08:41.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.868+0000 7fe28bfff640 10 client.4580.objecter successfully canceled tick 2024-03-29T11:08:41.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.868+0000 7fe28bfff640 1 -- 172.21.15.71:0/4201798576 >> v1:172.21.15.137:6816/973332665 conn(0x7fe27806deb0 legacy=0x7fe278070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:41.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.868+0000 7fe28bfff640 10 monclient: shutdown 2024-03-29T11:08:41.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.868+0000 7fe28bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:41.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.868+0000 7fe28bfff640 1 -- 172.21.15.71:0/4201798576 >> v1:172.21.15.71:6790/0 conn(0x7fe28c144fc0 legacy=0x7fe28c137290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:41.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.868+0000 7fe28bfff640 1 -- 172.21.15.71:0/4201798576 shutdown_connections 2024-03-29T11:08:41.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.868+0000 7fe28bfff640 1 -- 172.21.15.71:0/4201798576 >> 172.21.15.71:0/4201798576 conn(0x7fe28c0b3a30 msgr2=0x7fe28c0aecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:41.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.868+0000 7fe28bfff640 1 -- 172.21.15.71:0/4201798576 shutdown_connections 2024-03-29T11:08:41.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.868+0000 7fe28bfff640 1 -- 172.21.15.71:0/4201798576 wait complete. 2024-03-29T11:08:41.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:41.869+0000 7fe28bfff640 1 librados: shutdown 2024-03-29T11:08:41.863 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:08:48.671 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:08:48.818 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.838+0000 7f59c5825640 10 monclient: get_monmap_and_config 2024-03-29T11:08:48.818 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.838+0000 7f59c5825640 10 monclient: build_initial_monmap 2024-03-29T11:08:48.818 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.838+0000 7f59c5825640 10 monclient: monmap: 2024-03-29T11:08:48.818 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:08:48.818 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:48.818 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:08:48.839466+0000 2024-03-29T11:08:48.818 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:08:48.839466+0000 2024-03-29T11:08:48.819 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:48.819 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:48.819 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:08:48.819 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:08:48.819 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:08:48.819 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:48.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.839+0000 7f59c5825640 1 Processor -- start 2024-03-29T11:08:48.820 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.840+0000 7f59c5825640 1 -- start start 2024-03-29T11:08:48.820 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.840+0000 7f59c5825640 10 monclient: init 2024-03-29T11:08:48.820 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.840+0000 7f59c5825640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:48.820 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.840+0000 7f59c5825640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:08:48.820 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.840+0000 7f59c5825640 10 monclient(hunting): picked mon.noname-c con 0x7f59c00a9ac0 addr v1:172.21.15.137:6789/0 2024-03-29T11:08:48.820 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.841+0000 7f59c5825640 10 monclient(hunting): picked mon.noname-b con 0x7f59c013ee90 addr v1:172.21.15.71:6790/0 2024-03-29T11:08:48.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.841+0000 7f59c5825640 10 monclient(hunting): picked mon.noname-a con 0x7f59c00af900 addr v1:172.21.15.71:6789/0 2024-03-29T11:08:48.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.841+0000 7f59c5825640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c00afd10 con 0x7f59c00af900 2024-03-29T11:08:48.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.841+0000 7f59c5825640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c00af6f0 con 0x7f59c00a9ac0 2024-03-29T11:08:48.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.841+0000 7f59c5825640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c00aac80 con 0x7f59c013ee90 2024-03-29T11:08:48.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.841+0000 7f59c5825640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:48.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.841+0000 7f59c5825640 10 monclient(hunting): authenticate will time out at 1285.099182s 2024-03-29T11:08:48.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.841+0000 7f59be7fc640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f59c013ee90 0x7f59c013b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:50488/0 (socket says 172.21.15.71:50488) 2024-03-29T11:08:48.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.842+0000 7f59be7fc640 1 -- 172.21.15.71:0/1565638601 learned_addr learned my addr 172.21.15.71:0/1565638601 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:08:48.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.842+0000 7f59bdffb640 1 -- 172.21.15.71:0/1565638601 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205138830 0 0) 0x7f59c00aac80 con 0x7f59c013ee90 2024-03-29T11:08:48.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.842+0000 7f59bdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:08:48.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.842+0000 7f59bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:48.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.842+0000 7f59bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:48.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.842+0000 7f59bdffb640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:08:48.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.842+0000 7f59bdffb640 1 -- 172.21.15.71:0/1565638601 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59a8003520 con 0x7f59c013ee90 2024-03-29T11:08:48.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.842+0000 7f59bdffb640 1 -- 172.21.15.71:0/1565638601 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1927392830 0 0) 0x7f59a8003520 con 0x7f59c013ee90 2024-03-29T11:08:48.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.842+0000 7f59bdffb640 1 -- 172.21.15.71:0/1565638601 >> v1:172.21.15.137:6789/0 conn(0x7f59c00a9ac0 legacy=0x7f59c00adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:08:48.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.844+0000 7f59bdffb640 1 -- 172.21.15.71:0/1565638601 >> v1:172.21.15.71:6789/0 conn(0x7f59c00af900 legacy=0x7f59c0149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:08:48.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.844+0000 7f59bdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:08:48.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.844+0000 7f59bdffb640 1 monclient: found mon.noname-b 2024-03-29T11:08:48.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.844+0000 7f59bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:48.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.844+0000 7f59bdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:08:48.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.844+0000 7f59bdffb640 1 -- 172.21.15.71:0/1565638601 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c00aaf60 con 0x7f59c013ee90 2024-03-29T11:08:48.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.844+0000 7f59bdffb640 10 monclient: _finish_auth 0 2024-03-29T11:08:48.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.844+0000 7f59bdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:08:48.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.844+0000 7f59bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:48.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.844+0000 7f59bdffb640 1 -- 172.21.15.71:0/1565638601 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f59b4001670 con 0x7f59c013ee90 2024-03-29T11:08:48.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.844+0000 7f59bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:48.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.844+0000 7f59bdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:08:48.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.844+0000 7f59bdffb640 10 monclient: dump: 2024-03-29T11:08:48.829 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:48.829 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:48.829 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:48.829 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:48.829 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:48.829 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:48.829 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:48.829 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:48.829 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:48.829 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:48.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.844+0000 7f59c5825640 5 monclient: authenticate success, global_id 4583 2024-03-29T11:08:48.830 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.844+0000 7f59c5825640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:48.830 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.845+0000 7f59bdffb640 1 -- 172.21.15.71:0/1565638601 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59b40018d0 con 0x7f59c013ee90 2024-03-29T11:08:48.830 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.845+0000 7f59bdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:48.830 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.845+0000 7f59bdffb640 1 -- 172.21.15.71:0/1565638601 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f59b4001d20 con 0x7f59c013ee90 2024-03-29T11:08:48.830 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.845+0000 7f59bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:48.830 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.845+0000 7f59bdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:48.830 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.845+0000 7f59bdffb640 10 monclient: dump: 2024-03-29T11:08:48.830 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:48.830 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:48.830 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:48.830 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:48.830 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:48.830 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:48.830 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:48.831 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:48.831 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:48.831 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:48.831 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.845+0000 7f59c5825640 10 monclient: get_monmap_and_config success 2024-03-29T11:08:48.831 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.845+0000 7f59c5825640 4 set_mon_vals no callback set 2024-03-29T11:08:48.831 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.846+0000 7f59c5825640 10 monclient: shutdown 2024-03-29T11:08:48.831 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.846+0000 7f59c5825640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:48.831 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.846+0000 7f59c5825640 1 -- 172.21.15.71:0/1565638601 >> v1:172.21.15.71:6790/0 conn(0x7f59c013ee90 legacy=0x7f59c013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:48.831 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.846+0000 7f59c5825640 1 -- 172.21.15.71:0/1565638601 shutdown_connections 2024-03-29T11:08:48.831 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.846+0000 7f59c5825640 1 -- 172.21.15.71:0/1565638601 >> 172.21.15.71:0/1565638601 conn(0x7f59c00b35e0 msgr2=0x7f59c00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:48.831 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.847+0000 7f59c5825640 1 -- 172.21.15.71:0/1565638601 shutdown_connections 2024-03-29T11:08:48.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.847+0000 7f59c5825640 1 -- 172.21.15.71:0/1565638601 wait complete. 2024-03-29T11:08:48.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.848+0000 7f59c5825640 10 monclient: build_initial_monmap 2024-03-29T11:08:48.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.848+0000 7f59c5825640 10 monclient: monmap: 2024-03-29T11:08:48.832 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:08:48.832 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:08:48.832 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:08:48.832 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:08:48.833 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:48.833 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:48.833 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:08:48.833 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:08:48.833 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:08:48.833 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:48.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.848+0000 7f59c5825640 1 librados: starting msgr at 2024-03-29T11:08:48.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.848+0000 7f59c5825640 1 librados: starting objecter 2024-03-29T11:08:48.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.848+0000 7f59c5825640 1 Processor -- start 2024-03-29T11:08:48.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.848+0000 7f59c5825640 1 -- start start 2024-03-29T11:08:48.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.848+0000 7f59c5825640 1 librados: setting wanted keys 2024-03-29T11:08:48.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.848+0000 7f59c5825640 1 librados: calling monclient init 2024-03-29T11:08:48.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.848+0000 7f59c5825640 10 monclient: init 2024-03-29T11:08:48.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.849+0000 7f59c5825640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:48.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.849+0000 7f59c5825640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:08:48.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.852+0000 7f59c5825640 10 monclient(hunting): picked mon.noname-a con 0x7f59c00a9ac0 addr v1:172.21.15.71:6789/0 2024-03-29T11:08:48.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.852+0000 7f59c5825640 10 monclient(hunting): picked mon.noname-c con 0x7f59c00af900 addr v1:172.21.15.71:6790/0 2024-03-29T11:08:48.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.852+0000 7f59c5825640 10 monclient(hunting): picked mon.noname-b con 0x7f59c013ee90 addr v1:172.21.15.137:6789/0 2024-03-29T11:08:48.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.852+0000 7f59c5825640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c00713f0 con 0x7f59c00a9ac0 2024-03-29T11:08:48.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.852+0000 7f59c5825640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c0137580 con 0x7f59c013ee90 2024-03-29T11:08:48.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.852+0000 7f59c5825640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c0137860 con 0x7f59c00af900 2024-03-29T11:08:48.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.852+0000 7f59c5825640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:48.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.852+0000 7f59c5825640 10 monclient(hunting): authenticate will time out at 1285.110901s 2024-03-29T11:08:48.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.852+0000 7f59beffd640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f59c00a9ac0 0x7f59c0136120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:52231/0 (socket says 172.21.15.71:52231) 2024-03-29T11:08:48.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.852+0000 7f59beffd640 1 -- 172.21.15.71:0/462796127 learned_addr learned my addr 172.21.15.71:0/462796127 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:08:48.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.852+0000 7f59a7fff640 10 client.?.objecter ms_handle_connect 0x7f59c00af900 2024-03-29T11:08:48.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.852+0000 7f59a7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:48.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.853+0000 7f59a7fff640 1 -- 172.21.15.71:0/462796127 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3701913832 0 0) 0x7f59c0137860 con 0x7f59c00af900 2024-03-29T11:08:48.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.853+0000 7f59a7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:08:48.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.853+0000 7f59a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:48.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.853+0000 7f59a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:48.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.853+0000 7f59a7fff640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:08:48.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.853+0000 7f59a7fff640 1 -- 172.21.15.71:0/462796127 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59ac00d6c0 con 0x7f59c00af900 2024-03-29T11:08:48.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.853+0000 7f59a7fff640 1 -- 172.21.15.71:0/462796127 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2466485024 0 0) 0x7f59ac00d6c0 con 0x7f59c00af900 2024-03-29T11:08:48.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.853+0000 7f59a7fff640 1 -- 172.21.15.71:0/462796127 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59c0137860 con 0x7f59c00af900 2024-03-29T11:08:48.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.853+0000 7f59a7fff640 1 -- 172.21.15.71:0/462796127 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f59b4001630 con 0x7f59c00af900 2024-03-29T11:08:48.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.853+0000 7f59a7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:48.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.853+0000 7f59a7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:08:48.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.853+0000 7f59a7fff640 10 monclient(hunting): dump: 2024-03-29T11:08:48.837 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:48.837 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:48.837 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:48.837 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:48.837 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:48.837 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:48.837 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:48.837 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:48.837 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:48.837 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:48.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.853+0000 7f59a7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:08:48.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.853+0000 7f59a7fff640 1 -- 172.21.15.71:0/462796127 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1782425399 0 0) 0x7f59c0137860 con 0x7f59c00af900 2024-03-29T11:08:48.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.854+0000 7f59a7fff640 1 -- 172.21.15.71:0/462796127 >> v1:172.21.15.137:6789/0 conn(0x7f59c013ee90 legacy=0x7f59c00704d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:08:48.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.855+0000 7f59a7fff640 1 -- 172.21.15.71:0/462796127 >> v1:172.21.15.71:6789/0 conn(0x7f59c00a9ac0 legacy=0x7f59c0136120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:48.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.855+0000 7f59a7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:08:48.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.855+0000 7f59a7fff640 1 monclient: found mon.c 2024-03-29T11:08:48.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.855+0000 7f59a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:48.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.855+0000 7f59a7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:08:48.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.855+0000 7f59a7fff640 1 -- 172.21.15.71:0/462796127 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c0134f60 con 0x7f59c00af900 2024-03-29T11:08:48.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.856+0000 7f59a7fff640 10 monclient: _finish_auth 0 2024-03-29T11:08:48.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.856+0000 7f59a7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:08:48.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.856+0000 7f59a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:48.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.856+0000 7f59c5825640 5 monclient: authenticate success, global_id 4586 2024-03-29T11:08:48.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.856+0000 7f59c5825640 10 monclient: _renew_subs 2024-03-29T11:08:48.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.856+0000 7f59c5825640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:08:48.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.856+0000 7f59c5825640 1 -- 172.21.15.71:0/462796127 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c01351b0 con 0x7f59c00af900 2024-03-29T11:08:48.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.856+0000 7f59c5825640 10 client.4586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:08:48.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.856+0000 7f59c5825640 10 monclient: _renew_subs 2024-03-29T11:08:48.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.856+0000 7f59c5825640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:08:48.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.856+0000 7f59c5825640 1 -- 172.21.15.71:0/462796127 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c0138140 con 0x7f59c00af900 2024-03-29T11:08:48.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.856+0000 7f59c5825640 1 librados: init done 2024-03-29T11:08:48.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.856+0000 7f59a7fff640 1 -- 172.21.15.71:0/462796127 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59b4004a90 con 0x7f59c00af900 2024-03-29T11:08:48.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.856+0000 7f59a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:48.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.856+0000 7f59bd7fa640 4 set_mon_vals no callback set 2024-03-29T11:08:48.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.858+0000 7f59a7fff640 1 -- 172.21.15.71:0/462796127 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f59b4004e20 con 0x7f59c00af900 2024-03-29T11:08:48.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.858+0000 7f59a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:48.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.858+0000 7f59a7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:48.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.858+0000 7f59a7fff640 10 monclient: dump: 2024-03-29T11:08:48.841 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:48.841 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:48.841 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:48.841 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:48.842 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:48.842 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:48.842 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:48.842 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:48.842 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:48.842 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:48.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.858+0000 7f59a7fff640 1 -- 172.21.15.71:0/462796127 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f59b4005040 con 0x7f59c00af900 2024-03-29T11:08:48.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.858+0000 7f59c5825640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:48.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.858+0000 7f59c5825640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:48.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.858+0000 7f59c5825640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:08:48.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.858+0000 7f59c5825640 1 -- 172.21.15.71:0/462796127 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c013bd00 con 0x7f59c00af900 2024-03-29T11:08:48.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.858+0000 7f59a7fff640 1 -- 172.21.15.71:0/462796127 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f59b4084c10 con 0x7f59c00af900 2024-03-29T11:08:48.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.858+0000 7f59a7fff640 10 client.4586.objecter ms_dispatch 0x7f59c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:08:48.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.858+0000 7f59a7fff640 3 client.4586.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:08:48.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.858+0000 7f59a7fff640 3 client.4586.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:08:48.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.858+0000 7f59a7fff640 20 client.4586.objecter dump_active .. 0 homeless 2024-03-29T11:08:48.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.859+0000 7f59beffd640 10 monclient: get_auth_request con 0x7f59ac076430 auth_method 0 2024-03-29T11:08:48.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.859+0000 7f59a7fff640 10 client.4586.objecter ms_handle_connect 0x7f59ac076430 2024-03-29T11:08:48.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.868+0000 7f59a7fff640 1 -- 172.21.15.71:0/462796127 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f59c013bd00 con 0x7f59c00af900 2024-03-29T11:08:48.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.868+0000 7f59a7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:48.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:48.868+0000 7f59a7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:08:49.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:49.190+0000 7f59c5825640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:49.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:49.190+0000 7f59c5825640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:49.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:49.190+0000 7f59c5825640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:08:49.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:49.190+0000 7f59c5825640 1 -- 172.21.15.71:0/462796127 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59c013bd00 con 0x7f59c00af900 2024-03-29T11:08:49.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:49.193+0000 7f59a7fff640 1 -- 172.21.15.71:0/462796127 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f59c013bd00 con 0x7f59c00af900 2024-03-29T11:08:49.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:49.193+0000 7f59a7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:49.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:49.193+0000 7f59a7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:08:49.172 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:08:49.172 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:08:49.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:49.196+0000 7f59c5825640 10 librados: watch_flush enter 2024-03-29T11:08:49.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:49.196+0000 7f59c5825640 10 librados: watch_flush exit 2024-03-29T11:08:49.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:49.196+0000 7f59c5825640 20 client.4586.objecter shutdown clearing up homeless session... 2024-03-29T11:08:49.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:49.196+0000 7f59c5825640 10 client.4586.objecter successfully canceled tick 2024-03-29T11:08:49.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:49.196+0000 7f59c5825640 1 -- 172.21.15.71:0/462796127 >> v1:172.21.15.137:6816/973332665 conn(0x7f59ac076430 legacy=0x7f59ac0788f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:49.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:49.196+0000 7f59c5825640 10 monclient: shutdown 2024-03-29T11:08:49.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:49.196+0000 7f59c5825640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:49.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:49.196+0000 7f59c5825640 1 -- 172.21.15.71:0/462796127 >> v1:172.21.15.71:6790/0 conn(0x7f59c00af900 legacy=0x7f59c006fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:49.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:49.196+0000 7f59c5825640 1 -- 172.21.15.71:0/462796127 shutdown_connections 2024-03-29T11:08:49.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:49.196+0000 7f59c5825640 1 -- 172.21.15.71:0/462796127 >> 172.21.15.71:0/462796127 conn(0x7f59c00b35e0 msgr2=0x7f59c00ad0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:49.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:49.198+0000 7f59c5825640 1 -- 172.21.15.71:0/462796127 shutdown_connections 2024-03-29T11:08:49.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:49.199+0000 7f59c5825640 1 -- 172.21.15.71:0/462796127 wait complete. 2024-03-29T11:08:49.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:49.199+0000 7f59c5825640 1 librados: shutdown 2024-03-29T11:08:49.195 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:08:56.002 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:08:56.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.170+0000 7f65767cf640 10 monclient: get_monmap_and_config 2024-03-29T11:08:56.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.170+0000 7f65767cf640 10 monclient: build_initial_monmap 2024-03-29T11:08:56.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.170+0000 7f65767cf640 10 monclient: monmap: 2024-03-29T11:08:56.150 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:08:56.150 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:56.150 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:08:56.172327+0000 2024-03-29T11:08:56.150 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:08:56.172327+0000 2024-03-29T11:08:56.150 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:56.150 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:56.150 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:08:56.151 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:08:56.151 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:08:56.151 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:56.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.172+0000 7f65767cf640 1 Processor -- start 2024-03-29T11:08:56.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.172+0000 7f65767cf640 1 -- start start 2024-03-29T11:08:56.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.172+0000 7f65767cf640 10 monclient: init 2024-03-29T11:08:56.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.173+0000 7f65767cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:56.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.173+0000 7f65767cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:08:56.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.173+0000 7f65767cf640 10 monclient(hunting): picked mon.noname-a con 0x7f65700b0860 addr v1:172.21.15.71:6789/0 2024-03-29T11:08:56.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.173+0000 7f65767cf640 10 monclient(hunting): picked mon.noname-b con 0x7f6570072450 addr v1:172.21.15.71:6790/0 2024-03-29T11:08:56.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.173+0000 7f65767cf640 10 monclient(hunting): picked mon.noname-c con 0x7f657013ee40 addr v1:172.21.15.137:6789/0 2024-03-29T11:08:56.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.173+0000 7f65767cf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65700ad840 con 0x7f65700b0860 2024-03-29T11:08:56.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.173+0000 7f65767cf640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657013c830 con 0x7f657013ee40 2024-03-29T11:08:56.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.173+0000 7f65767cf640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657013c9c0 con 0x7f6570072450 2024-03-29T11:08:56.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.173+0000 7f65767cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:56.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.173+0000 7f65767cf640 10 monclient(hunting): authenticate will time out at 1292.432629s 2024-03-29T11:08:56.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.173+0000 7f6574d6c640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f657013ee40 0x7f657013f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:34664/0 (socket says 172.21.15.71:34664) 2024-03-29T11:08:56.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.173+0000 7f6574d6c640 1 -- 172.21.15.71:0/3782296136 learned_addr learned my addr 172.21.15.71:0/3782296136 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:08:56.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.174+0000 7f656effd640 1 -- 172.21.15.71:0/3782296136 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173210621 0 0) 0x7f657013c830 con 0x7f657013ee40 2024-03-29T11:08:56.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.174+0000 7f656effd640 10 monclient(hunting): _negotiate 2024-03-29T11:08:56.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.174+0000 7f656effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:56.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.174+0000 7f656effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:56.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.174+0000 7f656effd640 10 monclient(hunting): my global_id is 4480 2024-03-29T11:08:56.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.174+0000 7f656effd640 1 -- 172.21.15.71:0/3782296136 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6558005730 con 0x7f657013ee40 2024-03-29T11:08:56.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.174+0000 7f656effd640 1 -- 172.21.15.71:0/3782296136 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481623637 0 0) 0x7f657013c9c0 con 0x7f6570072450 2024-03-29T11:08:56.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.174+0000 7f656effd640 10 monclient(hunting): _negotiate 2024-03-29T11:08:56.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.174+0000 7f656effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:56.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.174+0000 7f656effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:56.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.174+0000 7f656effd640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:08:56.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.174+0000 7f656effd640 1 -- 172.21.15.71:0/3782296136 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6558005a50 con 0x7f6570072450 2024-03-29T11:08:56.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.174+0000 7f656effd640 1 -- 172.21.15.71:0/3782296136 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 295016615 0 0) 0x7f65700ad840 con 0x7f65700b0860 2024-03-29T11:08:56.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.174+0000 7f656effd640 10 monclient(hunting): _negotiate 2024-03-29T11:08:56.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.174+0000 7f656effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:56.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.174+0000 7f656effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:56.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.174+0000 7f656effd640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:08:56.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.174+0000 7f656effd640 1 -- 172.21.15.71:0/3782296136 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6558005d90 con 0x7f65700b0860 2024-03-29T11:08:56.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.175+0000 7f656effd640 1 -- 172.21.15.71:0/3782296136 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 261357031 0 0) 0x7f6558005a50 con 0x7f6570072450 2024-03-29T11:08:56.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.175+0000 7f656effd640 1 -- 172.21.15.71:0/3782296136 >> v1:172.21.15.137:6789/0 conn(0x7f657013ee40 legacy=0x7f657013f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:56.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.175+0000 7f656effd640 1 -- 172.21.15.71:0/3782296136 >> v1:172.21.15.71:6789/0 conn(0x7f65700b0860 legacy=0x7f65700acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:56.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.175+0000 7f656effd640 10 monclient: _finish_hunting 0 2024-03-29T11:08:56.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.175+0000 7f656effd640 1 monclient: found mon.noname-b 2024-03-29T11:08:56.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.175+0000 7f656effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:56.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.175+0000 7f656effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:08:56.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.175+0000 7f656effd640 1 -- 172.21.15.71:0/3782296136 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657013f970 con 0x7f6570072450 2024-03-29T11:08:56.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.175+0000 7f656effd640 10 monclient: _finish_auth 0 2024-03-29T11:08:56.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.175+0000 7f656effd640 10 monclient: _check_auth_tickets 2024-03-29T11:08:56.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.175+0000 7f656effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:56.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.175+0000 7f656effd640 1 -- 172.21.15.71:0/3782296136 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f65640015f0 con 0x7f6570072450 2024-03-29T11:08:56.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.175+0000 7f656effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:56.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.175+0000 7f656effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:08:56.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.175+0000 7f656effd640 10 monclient: dump: 2024-03-29T11:08:56.157 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:56.157 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:56.157 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:56.157 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:56.157 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:56.158 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:56.158 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:56.158 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:56.158 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:56.158 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:56.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.175+0000 7f65767cf640 5 monclient: authenticate success, global_id 4589 2024-03-29T11:08:56.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.175+0000 7f65767cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:08:56.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.176+0000 7f656effd640 1 -- 172.21.15.71:0/3782296136 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65640017d0 con 0x7f6570072450 2024-03-29T11:08:56.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.176+0000 7f656effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:56.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.176+0000 7f656effd640 1 -- 172.21.15.71:0/3782296136 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f6564001c20 con 0x7f6570072450 2024-03-29T11:08:56.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.176+0000 7f65767cf640 10 monclient: get_monmap_and_config success 2024-03-29T11:08:56.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.176+0000 7f656effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:56.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.176+0000 7f656effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:56.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.176+0000 7f656effd640 10 monclient: dump: 2024-03-29T11:08:56.159 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:56.159 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:56.159 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:56.159 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:56.159 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:56.159 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:56.159 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:56.159 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:56.159 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:56.159 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:56.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.176+0000 7f65767cf640 4 set_mon_vals no callback set 2024-03-29T11:08:56.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.176+0000 7f65767cf640 10 monclient: shutdown 2024-03-29T11:08:56.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.176+0000 7f65767cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:56.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.177+0000 7f65767cf640 1 -- 172.21.15.71:0/3782296136 >> v1:172.21.15.71:6790/0 conn(0x7f6570072450 legacy=0x7f657013c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:56.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.177+0000 7f65767cf640 1 -- 172.21.15.71:0/3782296136 shutdown_connections 2024-03-29T11:08:56.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.177+0000 7f65767cf640 1 -- 172.21.15.71:0/3782296136 >> 172.21.15.71:0/3782296136 conn(0x7f657006b660 msgr2=0x7f657006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:56.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.177+0000 7f65767cf640 1 -- 172.21.15.71:0/3782296136 shutdown_connections 2024-03-29T11:08:56.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.177+0000 7f65767cf640 1 -- 172.21.15.71:0/3782296136 wait complete. 2024-03-29T11:08:56.165 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.177+0000 7f65767cf640 10 monclient: build_initial_monmap 2024-03-29T11:08:56.165 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.177+0000 7f65767cf640 10 monclient: monmap: 2024-03-29T11:08:56.166 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:08:56.166 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:08:56.166 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:08:56.166 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:08:56.166 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:56.166 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:56.166 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:08:56.166 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:08:56.166 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:08:56.166 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:56.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.178+0000 7f65767cf640 1 librados: starting msgr at 2024-03-29T11:08:56.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.178+0000 7f65767cf640 1 librados: starting objecter 2024-03-29T11:08:56.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.178+0000 7f65767cf640 1 Processor -- start 2024-03-29T11:08:56.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.178+0000 7f65767cf640 1 -- start start 2024-03-29T11:08:56.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.178+0000 7f65767cf640 1 librados: setting wanted keys 2024-03-29T11:08:56.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.178+0000 7f65767cf640 1 librados: calling monclient init 2024-03-29T11:08:56.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.178+0000 7f65767cf640 10 monclient: init 2024-03-29T11:08:56.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.178+0000 7f65767cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:56.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.178+0000 7f65767cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:08:56.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.178+0000 7f65767cf640 10 monclient(hunting): picked mon.noname-c con 0x7f6570072450 addr v1:172.21.15.71:6790/0 2024-03-29T11:08:56.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.178+0000 7f65767cf640 10 monclient(hunting): picked mon.noname-b con 0x7f65700b0860 addr v1:172.21.15.137:6789/0 2024-03-29T11:08:56.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.178+0000 7f65767cf640 10 monclient(hunting): picked mon.noname-a con 0x7f657013ee40 addr v1:172.21.15.71:6789/0 2024-03-29T11:08:56.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.178+0000 7f65767cf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65701430c0 con 0x7f657013ee40 2024-03-29T11:08:56.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.178+0000 7f65767cf640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65700b1420 con 0x7f65700b0860 2024-03-29T11:08:56.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.178+0000 7f65767cf640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6570141f70 con 0x7f6570072450 2024-03-29T11:08:56.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.178+0000 7f65767cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:56.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.178+0000 7f65767cf640 10 monclient(hunting): authenticate will time out at 1292.438095s 2024-03-29T11:08:56.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.179+0000 7f656ffff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f6570072450 0x7f65700b2c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:38458/0 (socket says 172.21.15.71:38458) 2024-03-29T11:08:56.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.179+0000 7f656ffff640 1 -- 172.21.15.71:0/4226459499 learned_addr learned my addr 172.21.15.71:0/4226459499 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:08:56.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.179+0000 7f656cff9640 10 client.?.objecter ms_handle_connect 0x7f657013ee40 2024-03-29T11:08:56.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.179+0000 7f656cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:56.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.179+0000 7f656cff9640 10 client.?.objecter ms_handle_connect 0x7f6570072450 2024-03-29T11:08:56.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.179+0000 7f656cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:56.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.179+0000 7f656cff9640 10 client.?.objecter ms_handle_connect 0x7f65700b0860 2024-03-29T11:08:56.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.179+0000 7f656cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:08:56.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.180+0000 7f656cff9640 1 -- 172.21.15.71:0/4226459499 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2791429218 0 0) 0x7f6570141f70 con 0x7f6570072450 2024-03-29T11:08:56.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.180+0000 7f656cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:08:56.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.180+0000 7f656cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:56.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.180+0000 7f656cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:56.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.180+0000 7f656cff9640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:08:56.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.180+0000 7f656cff9640 1 -- 172.21.15.71:0/4226459499 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f655c004eb0 con 0x7f6570072450 2024-03-29T11:08:56.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.180+0000 7f656cff9640 1 -- 172.21.15.71:0/4226459499 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999073583 0 0) 0x7f65700b1420 con 0x7f65700b0860 2024-03-29T11:08:56.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.180+0000 7f656cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:08:56.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.180+0000 7f656cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:56.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.180+0000 7f656cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:56.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.180+0000 7f656cff9640 10 monclient(hunting): my global_id is 4483 2024-03-29T11:08:56.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.180+0000 7f656cff9640 1 -- 172.21.15.71:0/4226459499 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6570141f70 con 0x7f65700b0860 2024-03-29T11:08:56.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.180+0000 7f656cff9640 1 -- 172.21.15.71:0/4226459499 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 497082456 0 0) 0x7f65701430c0 con 0x7f657013ee40 2024-03-29T11:08:56.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.180+0000 7f656cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:08:56.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.180+0000 7f656cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:08:56.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.180+0000 7f656cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:08:56.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.180+0000 7f656cff9640 10 monclient(hunting): my global_id is 4539 2024-03-29T11:08:56.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.180+0000 7f656cff9640 1 -- 172.21.15.71:0/4226459499 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f655c005360 con 0x7f657013ee40 2024-03-29T11:08:56.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.181+0000 7f656cff9640 1 -- 172.21.15.71:0/4226459499 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1561922877 0 0) 0x7f655c004eb0 con 0x7f6570072450 2024-03-29T11:08:56.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.181+0000 7f656cff9640 1 -- 172.21.15.71:0/4226459499 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65701430c0 con 0x7f6570072450 2024-03-29T11:08:56.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.181+0000 7f656cff9640 1 -- 172.21.15.71:0/4226459499 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f6560001a30 con 0x7f6570072450 2024-03-29T11:08:56.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.181+0000 7f656cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:56.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.181+0000 7f656cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:08:56.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.181+0000 7f656cff9640 10 monclient(hunting): dump: 2024-03-29T11:08:56.170 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:56.170 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:56.170 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:56.170 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:56.170 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:56.170 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:56.170 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:56.170 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:56.170 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:56.170 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:56.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.181+0000 7f656cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:08:56.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.181+0000 7f656cff9640 1 -- 172.21.15.71:0/4226459499 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1276428343 0 0) 0x7f6570141f70 con 0x7f65700b0860 2024-03-29T11:08:56.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.181+0000 7f656cff9640 1 -- 172.21.15.71:0/4226459499 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f655c004eb0 con 0x7f65700b0860 2024-03-29T11:08:56.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.181+0000 7f656cff9640 1 -- 172.21.15.71:0/4226459499 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f6564001ae0 con 0x7f65700b0860 2024-03-29T11:08:56.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.181+0000 7f656cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:56.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.181+0000 7f656cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:08:56.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.181+0000 7f656cff9640 10 monclient(hunting): dump: 2024-03-29T11:08:56.171 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:56.171 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:56.171 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:56.171 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:56.171 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:56.171 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:56.171 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:56.171 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:56.172 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:56.172 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:56.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.181+0000 7f656cff9640 1 -- 172.21.15.71:0/4226459499 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3601299856 0 0) 0x7f655c005360 con 0x7f657013ee40 2024-03-29T11:08:56.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.181+0000 7f656cff9640 1 -- 172.21.15.71:0/4226459499 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6570141f70 con 0x7f657013ee40 2024-03-29T11:08:56.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.181+0000 7f656cff9640 1 -- 172.21.15.71:0/4226459499 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f6568001ad0 con 0x7f657013ee40 2024-03-29T11:08:56.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.181+0000 7f656cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:56.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.181+0000 7f656cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:08:56.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.181+0000 7f656cff9640 10 monclient(hunting): dump: 2024-03-29T11:08:56.172 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:56.172 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:56.172 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:56.172 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:56.172 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:56.172 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:56.172 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:56.173 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:56.173 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:56.173 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:56.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.182+0000 7f656cff9640 1 -- 172.21.15.71:0/4226459499 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 590840056 0 0) 0x7f65701430c0 con 0x7f6570072450 2024-03-29T11:08:56.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.182+0000 7f656cff9640 1 -- 172.21.15.71:0/4226459499 >> v1:172.21.15.137:6789/0 conn(0x7f65700b0860 legacy=0x7f657013d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:56.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.182+0000 7f656cff9640 1 -- 172.21.15.71:0/4226459499 >> v1:172.21.15.71:6789/0 conn(0x7f657013ee40 legacy=0x7f65701de770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:56.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.182+0000 7f656cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:08:56.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.182+0000 7f656cff9640 1 monclient: found mon.c 2024-03-29T11:08:56.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.182+0000 7f656cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:08:56.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.182+0000 7f656cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:08:56.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.182+0000 7f656cff9640 1 -- 172.21.15.71:0/4226459499 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6570141970 con 0x7f6570072450 2024-03-29T11:08:56.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.182+0000 7f656cff9640 10 monclient: _finish_auth 0 2024-03-29T11:08:56.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.182+0000 7f656cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:08:56.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.182+0000 7f656cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:08:56.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.182+0000 7f65767cf640 5 monclient: authenticate success, global_id 4592 2024-03-29T11:08:56.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.182+0000 7f65767cf640 10 monclient: _renew_subs 2024-03-29T11:08:56.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.182+0000 7f65767cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:08:56.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.182+0000 7f65767cf640 1 -- 172.21.15.71:0/4226459499 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6570141bc0 con 0x7f6570072450 2024-03-29T11:08:56.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.182+0000 7f65767cf640 10 client.4592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:08:56.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.182+0000 7f65767cf640 10 monclient: _renew_subs 2024-03-29T11:08:56.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.182+0000 7f65767cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:08:56.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.182+0000 7f656cff9640 1 -- 172.21.15.71:0/4226459499 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6560001c70 con 0x7f6570072450 2024-03-29T11:08:56.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.182+0000 7f65767cf640 1 -- 172.21.15.71:0/4226459499 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f65700b3500 con 0x7f6570072450 2024-03-29T11:08:56.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.182+0000 7f65767cf640 1 librados: init done 2024-03-29T11:08:56.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.182+0000 7f656cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:08:56.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.182+0000 7f656cff9640 1 -- 172.21.15.71:0/4226459499 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f6560003a60 con 0x7f6570072450 2024-03-29T11:08:56.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.182+0000 7f656cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:08:56.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.182+0000 7f656cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:08:56.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.182+0000 7f656cff9640 10 monclient: dump: 2024-03-29T11:08:56.174 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:08:56.175 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:08:56.175 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:56.175 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:08:56.175 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:08:56.175 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:08:56.175 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:08:56.175 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:08:56.175 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:08:56.175 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:08:56.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.182+0000 7f656e7fc640 4 set_mon_vals no callback set 2024-03-29T11:08:56.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.183+0000 7f65767cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:56.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.184+0000 7f656cff9640 1 -- 172.21.15.71:0/4226459499 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f6560003bc0 con 0x7f6570072450 2024-03-29T11:08:56.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.184+0000 7f65767cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:56.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.184+0000 7f65767cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:08:56.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.184+0000 7f65767cf640 1 -- 172.21.15.71:0/4226459499 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65700b0c40 con 0x7f6570072450 2024-03-29T11:08:56.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.185+0000 7f656cff9640 1 -- 172.21.15.71:0/4226459499 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f6560084b70 con 0x7f6570072450 2024-03-29T11:08:56.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.185+0000 7f656cff9640 10 client.4592.objecter ms_dispatch 0x7f6570000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:08:56.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.185+0000 7f656cff9640 3 client.4592.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:08:56.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.185+0000 7f656cff9640 3 client.4592.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:08:56.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.185+0000 7f656cff9640 20 client.4592.objecter dump_active .. 0 homeless 2024-03-29T11:08:56.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.185+0000 7f656f7fe640 10 monclient: get_auth_request con 0x7f655c06e530 auth_method 0 2024-03-29T11:08:56.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.185+0000 7f656cff9640 10 client.4592.objecter ms_handle_connect 0x7f655c06e530 2024-03-29T11:08:56.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.196+0000 7f656cff9640 1 -- 172.21.15.71:0/4226459499 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6560050450 con 0x7f6570072450 2024-03-29T11:08:56.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.196+0000 7f656cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:08:56.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.196+0000 7f656cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:08:56.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.560+0000 7f65767cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:56.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.560+0000 7f65767cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:56.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.560+0000 7f65767cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:08:56.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.560+0000 7f65767cf640 1 -- 172.21.15.71:0/4226459499 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6570075f10 con 0x7f6570072450 2024-03-29T11:08:56.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.561+0000 7f656cff9640 1 -- 172.21.15.71:0/4226459499 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f6560052380 con 0x7f6570072450 2024-03-29T11:08:56.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.561+0000 7f656cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:08:56.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.561+0000 7f656cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:08:56.541 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:08:56.541 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:08:56.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.565+0000 7f65767cf640 10 librados: watch_flush enter 2024-03-29T11:08:56.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.565+0000 7f65767cf640 10 librados: watch_flush exit 2024-03-29T11:08:56.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.565+0000 7f65767cf640 20 client.4592.objecter shutdown clearing up homeless session... 2024-03-29T11:08:56.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.565+0000 7f65767cf640 10 client.4592.objecter successfully canceled tick 2024-03-29T11:08:56.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.565+0000 7f65767cf640 1 -- 172.21.15.71:0/4226459499 >> v1:172.21.15.137:6816/973332665 conn(0x7f655c06e530 legacy=0x7f655c0709f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:56.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.565+0000 7f65767cf640 10 monclient: shutdown 2024-03-29T11:08:56.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.565+0000 7f65767cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:08:56.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.565+0000 7f65767cf640 1 -- 172.21.15.71:0/4226459499 >> v1:172.21.15.71:6790/0 conn(0x7f6570072450 legacy=0x7f65700b2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:08:56.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.565+0000 7f65767cf640 1 -- 172.21.15.71:0/4226459499 shutdown_connections 2024-03-29T11:08:56.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.565+0000 7f65767cf640 1 -- 172.21.15.71:0/4226459499 >> 172.21.15.71:0/4226459499 conn(0x7f657006b660 msgr2=0x7f657006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:08:56.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.565+0000 7f65767cf640 1 -- 172.21.15.71:0/4226459499 shutdown_connections 2024-03-29T11:08:56.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.565+0000 7f65767cf640 1 -- 172.21.15.71:0/4226459499 wait complete. 2024-03-29T11:08:56.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:08:56.565+0000 7f65767cf640 1 librados: shutdown 2024-03-29T11:08:56.559 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:09:03.369 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:09:03.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.563+0000 7fa0fffd0640 10 monclient: get_monmap_and_config 2024-03-29T11:09:03.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.563+0000 7fa0fffd0640 10 monclient: build_initial_monmap 2024-03-29T11:09:03.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.563+0000 7fa0fffd0640 10 monclient: monmap: 2024-03-29T11:09:03.542 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:09:03.542 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:03.543 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:09:03.564592+0000 2024-03-29T11:09:03.543 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:09:03.564592+0000 2024-03-29T11:09:03.543 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:03.543 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:03.543 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:09:03.543 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:09:03.543 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:09:03.543 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:03.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.564+0000 7fa0fffd0640 1 Processor -- start 2024-03-29T11:09:03.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.565+0000 7fa0fffd0640 1 -- start start 2024-03-29T11:09:03.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.565+0000 7fa0fffd0640 10 monclient: init 2024-03-29T11:09:03.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.566+0000 7fa0fffd0640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:03.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.566+0000 7fa0fffd0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:09:03.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.566+0000 7fa0fffd0640 10 monclient(hunting): picked mon.noname-b con 0x7fa0f8070120 addr v1:172.21.15.71:6790/0 2024-03-29T11:09:03.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.566+0000 7fa0fffd0640 10 monclient(hunting): picked mon.noname-c con 0x7fa0f80b0420 addr v1:172.21.15.137:6789/0 2024-03-29T11:09:03.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.566+0000 7fa0fdd6c640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fa0f8070120 0x7fa0f813d350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:55492/0 (socket says 172.21.15.71:55492) 2024-03-29T11:09:03.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.566+0000 7fa0fffd0640 10 monclient(hunting): picked mon.noname-a con 0x7fa0f8070500 addr v1:172.21.15.71:6789/0 2024-03-29T11:09:03.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.566+0000 7fa0fffd0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f8072090 con 0x7fa0f8070500 2024-03-29T11:09:03.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.566+0000 7fa0fffd0640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f8072370 con 0x7fa0f80b0420 2024-03-29T11:09:03.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.566+0000 7fa0fffd0640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f8072650 con 0x7fa0f8070120 2024-03-29T11:09:03.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.566+0000 7fa0fffd0640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:03.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.566+0000 7fa0fffd0640 10 monclient(hunting): authenticate will time out at 1299.824972s 2024-03-29T11:09:03.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.567+0000 7fa0fdd6c640 1 -- 172.21.15.71:0/860937844 learned_addr learned my addr 172.21.15.71:0/860937844 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:09:03.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.570+0000 7fa0fcd6a640 1 -- 172.21.15.71:0/860937844 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1985288059 0 0) 0x7fa0f8072650 con 0x7fa0f8070120 2024-03-29T11:09:03.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.570+0000 7fa0fcd6a640 10 monclient(hunting): _negotiate 2024-03-29T11:09:03.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.570+0000 7fa0fcd6a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:03.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.570+0000 7fa0fcd6a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:03.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.570+0000 7fa0fcd6a640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:09:03.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 1 -- 172.21.15.71:0/860937844 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0e4003520 con 0x7fa0f8070120 2024-03-29T11:09:03.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 1 -- 172.21.15.71:0/860937844 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223631304 0 0) 0x7fa0f8072090 con 0x7fa0f8070500 2024-03-29T11:09:03.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 10 monclient(hunting): _negotiate 2024-03-29T11:09:03.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:03.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:03.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:09:03.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 1 -- 172.21.15.71:0/860937844 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0f8072650 con 0x7fa0f8070500 2024-03-29T11:09:03.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 1 -- 172.21.15.71:0/860937844 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2415714408 0 0) 0x7fa0e4003520 con 0x7fa0f8070120 2024-03-29T11:09:03.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 1 -- 172.21.15.71:0/860937844 >> v1:172.21.15.137:6789/0 conn(0x7fa0f80b0420 legacy=0x7fa0f81405b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:09:03.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 1 -- 172.21.15.71:0/860937844 >> v1:172.21.15.71:6789/0 conn(0x7fa0f8070500 legacy=0x7fa0f814b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:03.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 10 monclient: _finish_hunting 0 2024-03-29T11:09:03.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 1 monclient: found mon.noname-b 2024-03-29T11:09:03.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:03.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:09:03.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 1 -- 172.21.15.71:0/860937844 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f8140090 con 0x7fa0f8070120 2024-03-29T11:09:03.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 10 monclient: _finish_auth 0 2024-03-29T11:09:03.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 10 monclient: _check_auth_tickets 2024-03-29T11:09:03.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:03.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 1 -- 172.21.15.71:0/860937844 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fa0e8001690 con 0x7fa0f8070120 2024-03-29T11:09:03.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:03.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:09:03.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 10 monclient: dump: 2024-03-29T11:09:03.551 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:03.551 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:03.551 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:03.552 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:03.552 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:03.552 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:03.552 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:03.552 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:03.552 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:03.552 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:03.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fffd0640 5 monclient: authenticate success, global_id 4595 2024-03-29T11:09:03.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fffd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:03.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 1 -- 172.21.15.71:0/860937844 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0e80017f0 con 0x7fa0f8070120 2024-03-29T11:09:03.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:03.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 1 -- 172.21.15.71:0/860937844 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fa0e8001b80 con 0x7fa0f8070120 2024-03-29T11:09:03.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:03.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:03.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fcd6a640 10 monclient: dump: 2024-03-29T11:09:03.553 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:03.553 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:03.553 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:03.553 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:03.553 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:03.553 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:03.553 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:03.553 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:03.553 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:03.553 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:03.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fffd0640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:03.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.571+0000 7fa0fffd0640 4 set_mon_vals no callback set 2024-03-29T11:09:03.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.572+0000 7fa0fffd0640 10 monclient: shutdown 2024-03-29T11:09:03.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.572+0000 7fa0fffd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:03.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.572+0000 7fa0fffd0640 1 -- 172.21.15.71:0/860937844 >> v1:172.21.15.71:6790/0 conn(0x7fa0f8070120 legacy=0x7fa0f813d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:03.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.575+0000 7fa0fffd0640 1 -- 172.21.15.71:0/860937844 shutdown_connections 2024-03-29T11:09:03.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.575+0000 7fa0fffd0640 1 -- 172.21.15.71:0/860937844 >> 172.21.15.71:0/860937844 conn(0x7fa0f8067500 msgr2=0x7fa0f80b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:03.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.576+0000 7fa0fffd0640 1 -- 172.21.15.71:0/860937844 shutdown_connections 2024-03-29T11:09:03.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.577+0000 7fa0fffd0640 1 -- 172.21.15.71:0/860937844 wait complete. 2024-03-29T11:09:03.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.577+0000 7fa0fffd0640 10 monclient: build_initial_monmap 2024-03-29T11:09:03.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.577+0000 7fa0fffd0640 10 monclient: monmap: 2024-03-29T11:09:03.556 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:09:03.556 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:03.557 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:09:03.557 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:09:03.557 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:03.557 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:03.557 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:09:03.557 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:09:03.557 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:09:03.557 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:03.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.578+0000 7fa0fffd0640 1 librados: starting msgr at 2024-03-29T11:09:03.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.578+0000 7fa0fffd0640 1 librados: starting objecter 2024-03-29T11:09:03.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.578+0000 7fa0fffd0640 1 Processor -- start 2024-03-29T11:09:03.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.578+0000 7fa0fffd0640 1 -- start start 2024-03-29T11:09:03.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.578+0000 7fa0fffd0640 1 librados: setting wanted keys 2024-03-29T11:09:03.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.578+0000 7fa0fffd0640 1 librados: calling monclient init 2024-03-29T11:09:03.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.578+0000 7fa0fffd0640 10 monclient: init 2024-03-29T11:09:03.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.579+0000 7fa0fffd0640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:03.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.579+0000 7fa0fffd0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:09:03.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.579+0000 7fa0fffd0640 10 monclient(hunting): picked mon.noname-c con 0x7fa0f8070120 addr v1:172.21.15.71:6790/0 2024-03-29T11:09:03.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.579+0000 7fa0fffd0640 10 monclient(hunting): picked mon.noname-a con 0x7fa0f8070500 addr v1:172.21.15.71:6789/0 2024-03-29T11:09:03.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.579+0000 7fa0fffd0640 10 monclient(hunting): picked mon.noname-b con 0x7fa0f80b0420 addr v1:172.21.15.137:6789/0 2024-03-29T11:09:03.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.579+0000 7fa0fffd0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f814b030 con 0x7fa0f8070500 2024-03-29T11:09:03.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.579+0000 7fa0fffd0640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f8132560 con 0x7fa0f80b0420 2024-03-29T11:09:03.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.579+0000 7fa0fffd0640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f814be10 con 0x7fa0f8070120 2024-03-29T11:09:03.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.579+0000 7fa0fffd0640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:03.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.579+0000 7fa0fffd0640 10 monclient(hunting): authenticate will time out at 1299.837551s 2024-03-29T11:09:03.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.580+0000 7fa0fdd6c640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fa0f8070120 0x7fa0f806bc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:59938/0 (socket says 172.21.15.71:59938) 2024-03-29T11:09:03.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.580+0000 7fa0fdd6c640 1 -- 172.21.15.71:0/1013366011 learned_addr learned my addr 172.21.15.71:0/1013366011 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:09:03.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.580+0000 7fa0ee7fc640 10 client.?.objecter ms_handle_connect 0x7fa0f8070120 2024-03-29T11:09:03.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.580+0000 7fa0ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:03.560 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.580+0000 7fa0ee7fc640 1 -- 172.21.15.71:0/1013366011 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307708396 0 0) 0x7fa0f814be10 con 0x7fa0f8070120 2024-03-29T11:09:03.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.581+0000 7fa0ee7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:09:03.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.581+0000 7fa0ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:03.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.581+0000 7fa0ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:03.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.581+0000 7fa0ee7fc640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:09:03.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.581+0000 7fa0ee7fc640 1 -- 172.21.15.71:0/1013366011 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0dc004eb0 con 0x7fa0f8070120 2024-03-29T11:09:03.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.582+0000 7fa0ee7fc640 10 client.?.objecter ms_handle_connect 0x7fa0f8070500 2024-03-29T11:09:03.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.582+0000 7fa0ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:03.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.582+0000 7fa0ee7fc640 1 -- 172.21.15.71:0/1013366011 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 721931812 0 0) 0x7fa0dc004eb0 con 0x7fa0f8070120 2024-03-29T11:09:03.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.582+0000 7fa0ee7fc640 1 -- 172.21.15.71:0/1013366011 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0f814be10 con 0x7fa0f8070120 2024-03-29T11:09:03.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.582+0000 7fa0ee7fc640 1 -- 172.21.15.71:0/1013366011 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fa0e8001bc0 con 0x7fa0f8070120 2024-03-29T11:09:03.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.582+0000 7fa0ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:03.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.582+0000 7fa0ee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:09:03.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.582+0000 7fa0ee7fc640 10 monclient(hunting): dump: 2024-03-29T11:09:03.562 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:03.562 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:03.562 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:03.562 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:03.562 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:03.562 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:03.562 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:03.562 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:03.562 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:03.562 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:03.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.582+0000 7fa0ee7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:03.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.583+0000 7fa0ee7fc640 1 -- 172.21.15.71:0/1013366011 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1117020240 0 0) 0x7fa0f814b030 con 0x7fa0f8070500 2024-03-29T11:09:03.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.583+0000 7fa0ee7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:09:03.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.583+0000 7fa0ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:03.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.583+0000 7fa0ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:03.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.583+0000 7fa0ee7fc640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:09:03.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.583+0000 7fa0ee7fc640 1 -- 172.21.15.71:0/1013366011 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0dc004eb0 con 0x7fa0f8070500 2024-03-29T11:09:03.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.583+0000 7fa0ee7fc640 1 -- 172.21.15.71:0/1013366011 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4012676617 0 0) 0x7fa0f814be10 con 0x7fa0f8070120 2024-03-29T11:09:03.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.583+0000 7fa0ee7fc640 1 -- 172.21.15.71:0/1013366011 >> v1:172.21.15.137:6789/0 conn(0x7fa0f80b0420 legacy=0x7fa0f8131e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:03.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.583+0000 7fa0ee7fc640 1 -- 172.21.15.71:0/1013366011 >> v1:172.21.15.71:6789/0 conn(0x7fa0f8070500 legacy=0x7fa0f8131740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:03.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.583+0000 7fa0ee7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:09:03.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.583+0000 7fa0ee7fc640 1 monclient: found mon.c 2024-03-29T11:09:03.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.583+0000 7fa0ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:03.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.583+0000 7fa0ee7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:03.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.583+0000 7fa0ee7fc640 1 -- 172.21.15.71:0/1013366011 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f814c0f0 con 0x7fa0f8070120 2024-03-29T11:09:03.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.583+0000 7fa0ee7fc640 10 monclient: _finish_auth 0 2024-03-29T11:09:03.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.583+0000 7fa0ee7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:09:03.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.583+0000 7fa0ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:03.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.583+0000 7fa0fffd0640 5 monclient: authenticate success, global_id 4598 2024-03-29T11:09:03.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.583+0000 7fa0fffd0640 10 monclient: _renew_subs 2024-03-29T11:09:03.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.583+0000 7fa0fffd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:03.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.583+0000 7fa0fffd0640 1 -- 172.21.15.71:0/1013366011 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f814c3d0 con 0x7fa0f8070120 2024-03-29T11:09:03.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.584+0000 7fa0fffd0640 10 client.4598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:03.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.584+0000 7fa0fffd0640 10 monclient: _renew_subs 2024-03-29T11:09:03.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.584+0000 7fa0fffd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:03.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.584+0000 7fa0fffd0640 1 -- 172.21.15.71:0/1013366011 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f806c390 con 0x7fa0f8070120 2024-03-29T11:09:03.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.584+0000 7fa0fffd0640 1 librados: init done 2024-03-29T11:09:03.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.584+0000 7fa0ee7fc640 1 -- 172.21.15.71:0/1013366011 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0e8004a90 con 0x7fa0f8070120 2024-03-29T11:09:03.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.584+0000 7fa0ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:03.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.584+0000 7fa0ee7fc640 1 -- 172.21.15.71:0/1013366011 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fa0e8004e60 con 0x7fa0f8070120 2024-03-29T11:09:03.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.584+0000 7fa0ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:03.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.584+0000 7fa0ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:03.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.584+0000 7fa0ee7fc640 10 monclient: dump: 2024-03-29T11:09:03.567 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:03.567 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:03.567 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:03.567 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:03.567 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:03.567 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:03.567 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:03.567 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:03.567 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:03.567 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:03.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.586+0000 7fa0fffd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:03.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.586+0000 7fa0fffd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:03.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.586+0000 7fa0fffd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:03.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.586+0000 7fa0fffd0640 1 -- 172.21.15.71:0/1013366011 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f806c690 con 0x7fa0f8070120 2024-03-29T11:09:03.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.587+0000 7fa0effff640 4 set_mon_vals no callback set 2024-03-29T11:09:03.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.592+0000 7fa0ee7fc640 1 -- 172.21.15.71:0/1013366011 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7fa0e8004fc0 con 0x7fa0f8070120 2024-03-29T11:09:03.574 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.593+0000 7fa0ee7fc640 1 -- 172.21.15.71:0/1013366011 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7fa0e8084c70 con 0x7fa0f8070120 2024-03-29T11:09:03.574 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.593+0000 7fa0ee7fc640 10 client.4598.objecter ms_dispatch 0x7fa0f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:09:03.574 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.593+0000 7fa0ee7fc640 3 client.4598.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:09:03.574 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.593+0000 7fa0ee7fc640 3 client.4598.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:09:03.574 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.593+0000 7fa0ee7fc640 20 client.4598.objecter dump_active .. 0 homeless 2024-03-29T11:09:03.574 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.593+0000 7fa0fd56b640 10 monclient: get_auth_request con 0x7fa0dc06de00 auth_method 0 2024-03-29T11:09:03.574 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.594+0000 7fa0ee7fc640 10 client.4598.objecter ms_handle_connect 0x7fa0dc06de00 2024-03-29T11:09:03.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.599+0000 7fa0ee7fc640 1 -- 172.21.15.71:0/1013366011 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa0e8050550 con 0x7fa0f8070120 2024-03-29T11:09:03.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.599+0000 7fa0ee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:03.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.599+0000 7fa0ee7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:03.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.950+0000 7fa0fffd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:03.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.950+0000 7fa0fffd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:03.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.950+0000 7fa0fffd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:03.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.950+0000 7fa0fffd0640 1 -- 172.21.15.71:0/1013366011 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0f8075f10 con 0x7fa0f8070120 2024-03-29T11:09:03.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.954+0000 7fa0ee7fc640 1 -- 172.21.15.71:0/1013366011 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7fa0e8052480 con 0x7fa0f8070120 2024-03-29T11:09:03.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.954+0000 7fa0ee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:03.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.954+0000 7fa0ee7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:03.933 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:09:03.933 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:09:03.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.958+0000 7fa0fffd0640 10 librados: watch_flush enter 2024-03-29T11:09:03.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.958+0000 7fa0fffd0640 10 librados: watch_flush exit 2024-03-29T11:09:03.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.958+0000 7fa0fffd0640 20 client.4598.objecter shutdown clearing up homeless session... 2024-03-29T11:09:03.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.958+0000 7fa0fffd0640 10 client.4598.objecter successfully canceled tick 2024-03-29T11:09:03.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.958+0000 7fa0fffd0640 1 -- 172.21.15.71:0/1013366011 >> v1:172.21.15.137:6816/973332665 conn(0x7fa0dc06de00 legacy=0x7fa0dc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:03.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.958+0000 7fa0fffd0640 10 monclient: shutdown 2024-03-29T11:09:03.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.958+0000 7fa0fffd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:03.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.958+0000 7fa0fffd0640 1 -- 172.21.15.71:0/1013366011 >> v1:172.21.15.71:6790/0 conn(0x7fa0f8070120 legacy=0x7fa0f806bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:03.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.960+0000 7fa0fffd0640 1 -- 172.21.15.71:0/1013366011 shutdown_connections 2024-03-29T11:09:03.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.960+0000 7fa0fffd0640 1 -- 172.21.15.71:0/1013366011 >> 172.21.15.71:0/1013366011 conn(0x7fa0f8067500 msgr2=0x7fa0f81405b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:03.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.962+0000 7fa0fffd0640 1 -- 172.21.15.71:0/1013366011 shutdown_connections 2024-03-29T11:09:03.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.962+0000 7fa0fffd0640 1 -- 172.21.15.71:0/1013366011 wait complete. 2024-03-29T11:09:03.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:03.964+0000 7fa0fffd0640 1 librados: shutdown 2024-03-29T11:09:03.959 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:09:10.767 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:09:10.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.935+0000 7fafdbfff640 10 monclient: get_monmap_and_config 2024-03-29T11:09:10.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.935+0000 7fafdbfff640 10 monclient: build_initial_monmap 2024-03-29T11:09:10.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.935+0000 7fafdbfff640 10 monclient: monmap: 2024-03-29T11:09:10.914 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:09:10.914 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:10.914 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:09:10.936207+0000 2024-03-29T11:09:10.914 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:09:10.936207+0000 2024-03-29T11:09:10.914 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:10.914 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:10.914 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:09:10.914 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:09:10.914 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:09:10.915 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:10.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.936+0000 7fafdbfff640 1 Processor -- start 2024-03-29T11:09:10.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.937+0000 7fafdbfff640 1 -- start start 2024-03-29T11:09:10.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.937+0000 7fafdbfff640 10 monclient: init 2024-03-29T11:09:10.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.937+0000 7fafdbfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:10.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.937+0000 7fafdbfff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:09:10.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.937+0000 7fafdbfff640 10 monclient(hunting): picked mon.noname-a con 0x7fafdc0b4460 addr v1:172.21.15.71:6789/0 2024-03-29T11:09:10.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.937+0000 7fafdbfff640 10 monclient(hunting): picked mon.noname-b con 0x7fafdc0a9e00 addr v1:172.21.15.71:6790/0 2024-03-29T11:09:10.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.937+0000 7fafdbfff640 10 monclient(hunting): picked mon.noname-c con 0x7fafdc13f900 addr v1:172.21.15.137:6789/0 2024-03-29T11:09:10.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.937+0000 7fafdbfff640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafdc0ab0d0 con 0x7fafdc0b4460 2024-03-29T11:09:10.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.937+0000 7fafdbfff640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafdc0a9b40 con 0x7fafdc13f900 2024-03-29T11:09:10.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.938+0000 7fafdbfff640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafdc0b4da0 con 0x7fafdc0a9e00 2024-03-29T11:09:10.918 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.938+0000 7fafdbfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:10.918 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.938+0000 7fafdbfff640 10 monclient(hunting): authenticate will time out at 1307.197105s 2024-03-29T11:09:10.918 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.939+0000 7fafda7fc640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fafdc0a9e00 0x7fafdc142230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:47858/0 (socket says 172.21.15.71:47858) 2024-03-29T11:09:10.918 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.939+0000 7fafda7fc640 1 -- 172.21.15.71:0/2063678150 learned_addr learned my addr 172.21.15.71:0/2063678150 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:09:10.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.941+0000 7fafd9ffb640 1 -- 172.21.15.71:0/2063678150 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1558228907 0 0) 0x7fafdc0b4da0 con 0x7fafdc0a9e00 2024-03-29T11:09:10.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.941+0000 7fafd9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:09:10.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.941+0000 7fafd9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:10.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.941+0000 7fafd9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:10.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.941+0000 7fafd9ffb640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:09:10.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.941+0000 7fafd9ffb640 1 -- 172.21.15.71:0/2063678150 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafc4003520 con 0x7fafdc0a9e00 2024-03-29T11:09:10.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.942+0000 7fafd9ffb640 1 -- 172.21.15.71:0/2063678150 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 563584943 0 0) 0x7fafc4003520 con 0x7fafdc0a9e00 2024-03-29T11:09:10.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.942+0000 7fafd9ffb640 1 -- 172.21.15.71:0/2063678150 >> v1:172.21.15.137:6789/0 conn(0x7fafdc13f900 legacy=0x7fafdc142940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:10.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.942+0000 7fafd9ffb640 1 -- 172.21.15.71:0/2063678150 >> v1:172.21.15.71:6789/0 conn(0x7fafdc0b4460 legacy=0x7fafdc13f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:10.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.942+0000 7fafd9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:09:10.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.942+0000 7fafd9ffb640 1 monclient: found mon.noname-b 2024-03-29T11:09:10.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.942+0000 7fafd9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:10.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.942+0000 7fafd9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:09:10.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.942+0000 7fafd9ffb640 1 -- 172.21.15.71:0/2063678150 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafdc0b4840 con 0x7fafdc0a9e00 2024-03-29T11:09:10.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.942+0000 7fafd9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:09:10.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.942+0000 7fafd9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:09:10.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.942+0000 7fafd9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:10.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.942+0000 7fafd9ffb640 1 -- 172.21.15.71:0/2063678150 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fafcc001690 con 0x7fafdc0a9e00 2024-03-29T11:09:10.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.942+0000 7fafd9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:10.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.942+0000 7fafd9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:09:10.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.942+0000 7fafd9ffb640 10 monclient: dump: 2024-03-29T11:09:10.924 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:10.924 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:10.924 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:10.924 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:10.924 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:10.924 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:10.924 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:10.924 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:10.924 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:10.925 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:10.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.942+0000 7fafd9ffb640 1 -- 172.21.15.71:0/2063678150 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafcc0018b0 con 0x7fafdc0a9e00 2024-03-29T11:09:10.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.942+0000 7fafd9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:10.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.942+0000 7fafd9ffb640 1 -- 172.21.15.71:0/2063678150 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fafcc001c40 con 0x7fafdc0a9e00 2024-03-29T11:09:10.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.942+0000 7fafd9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:10.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.942+0000 7fafd9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:10.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.942+0000 7fafd9ffb640 10 monclient: dump: 2024-03-29T11:09:10.925 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:10.925 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:10.925 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:10.926 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:10.926 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:10.926 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:10.926 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:10.926 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:10.926 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:10.926 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:10.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.942+0000 7fafdbfff640 5 monclient: authenticate success, global_id 4601 2024-03-29T11:09:10.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.942+0000 7fafdbfff640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:10.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.942+0000 7fafdbfff640 4 set_mon_vals no callback set 2024-03-29T11:09:10.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.943+0000 7fafdbfff640 10 monclient: shutdown 2024-03-29T11:09:10.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.943+0000 7fafdbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:10.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.943+0000 7fafdbfff640 1 -- 172.21.15.71:0/2063678150 >> v1:172.21.15.71:6790/0 conn(0x7fafdc0a9e00 legacy=0x7fafdc142230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:10.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.944+0000 7fafdbfff640 1 -- 172.21.15.71:0/2063678150 shutdown_connections 2024-03-29T11:09:10.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.944+0000 7fafdbfff640 1 -- 172.21.15.71:0/2063678150 >> 172.21.15.71:0/2063678150 conn(0x7fafdc06c980 msgr2=0x7fafdc06cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:10.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.945+0000 7fafdbfff640 1 -- 172.21.15.71:0/2063678150 shutdown_connections 2024-03-29T11:09:10.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.945+0000 7fafdbfff640 1 -- 172.21.15.71:0/2063678150 wait complete. 2024-03-29T11:09:10.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.946+0000 7fafdbfff640 10 monclient: build_initial_monmap 2024-03-29T11:09:10.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.946+0000 7fafdbfff640 10 monclient: monmap: 2024-03-29T11:09:10.929 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:09:10.929 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:10.929 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:09:10.929 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:09:10.930 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:10.930 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:10.930 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:09:10.930 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:09:10.930 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:09:10.930 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:10.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.946+0000 7fafdbfff640 1 librados: starting msgr at 2024-03-29T11:09:10.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.946+0000 7fafdbfff640 1 librados: starting objecter 2024-03-29T11:09:10.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.947+0000 7fafdbfff640 1 Processor -- start 2024-03-29T11:09:10.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.947+0000 7fafdbfff640 1 -- start start 2024-03-29T11:09:10.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.947+0000 7fafdbfff640 1 librados: setting wanted keys 2024-03-29T11:09:10.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.947+0000 7fafdbfff640 1 librados: calling monclient init 2024-03-29T11:09:10.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.947+0000 7fafdbfff640 10 monclient: init 2024-03-29T11:09:10.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.948+0000 7fafdbfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:10.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.948+0000 7fafdbfff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:09:10.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.948+0000 7fafdbfff640 10 monclient(hunting): picked mon.noname-c con 0x7fafdc0a9e00 addr v1:172.21.15.71:6790/0 2024-03-29T11:09:10.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.948+0000 7fafdbfff640 10 monclient(hunting): picked mon.noname-a con 0x7fafdc0b4460 addr v1:172.21.15.71:6789/0 2024-03-29T11:09:10.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.948+0000 7fafdbfff640 10 monclient(hunting): picked mon.noname-b con 0x7fafdc13f900 addr v1:172.21.15.137:6789/0 2024-03-29T11:09:10.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.948+0000 7fafdbfff640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafdc0ae3f0 con 0x7fafdc0b4460 2024-03-29T11:09:10.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.948+0000 7fafdbfff640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafdc0ae6d0 con 0x7fafdc13f900 2024-03-29T11:09:10.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.948+0000 7fafdbfff640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafdc0ae9b0 con 0x7fafdc0a9e00 2024-03-29T11:09:10.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.948+0000 7fafdbfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:10.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.948+0000 7fafdbfff640 10 monclient(hunting): authenticate will time out at 1307.206960s 2024-03-29T11:09:10.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.951+0000 7fafdaffd640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fafdc0a9e00 0x7fafdc0708b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:48980/0 (socket says 172.21.15.71:48980) 2024-03-29T11:09:10.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.951+0000 7fafdaffd640 1 -- 172.21.15.71:0/1989406241 learned_addr learned my addr 172.21.15.71:0/1989406241 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:09:10.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.951+0000 7fafc37fe640 10 client.?.objecter ms_handle_connect 0x7fafdc0a9e00 2024-03-29T11:09:10.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.951+0000 7fafc37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:10.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.952+0000 7fafc37fe640 1 -- 172.21.15.71:0/1989406241 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297648403 0 0) 0x7fafdc0ae9b0 con 0x7fafdc0a9e00 2024-03-29T11:09:10.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.952+0000 7fafc37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:09:10.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.952+0000 7fafc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:10.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.952+0000 7fafc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:10.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.952+0000 7fafc37fe640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:09:10.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.952+0000 7fafc37fe640 1 -- 172.21.15.71:0/1989406241 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafc8004eb0 con 0x7fafdc0a9e00 2024-03-29T11:09:10.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.952+0000 7fafc37fe640 10 client.?.objecter ms_handle_connect 0x7fafdc13f900 2024-03-29T11:09:10.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.952+0000 7fafc37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:10.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.952+0000 7fafc37fe640 10 client.?.objecter ms_handle_connect 0x7fafdc0b4460 2024-03-29T11:09:10.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.952+0000 7fafc37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:10.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.952+0000 7fafc37fe640 1 -- 172.21.15.71:0/1989406241 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3116280663 0 0) 0x7fafc8004eb0 con 0x7fafdc0a9e00 2024-03-29T11:09:10.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.952+0000 7fafc37fe640 1 -- 172.21.15.71:0/1989406241 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafdc0ae9b0 con 0x7fafdc0a9e00 2024-03-29T11:09:10.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.952+0000 7fafc37fe640 1 -- 172.21.15.71:0/1989406241 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fafd0001990 con 0x7fafdc0a9e00 2024-03-29T11:09:10.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.952+0000 7fafc37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:10.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.952+0000 7fafc37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:09:10.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.952+0000 7fafc37fe640 10 monclient(hunting): dump: 2024-03-29T11:09:10.938 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:10.938 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:10.938 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:10.938 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:10.938 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:10.938 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:10.938 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:10.938 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:10.938 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:10.938 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:10.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.952+0000 7fafc37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:10.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafc37fe640 1 -- 172.21.15.71:0/1989406241 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489470129 0 0) 0x7fafdc0ae3f0 con 0x7fafdc0b4460 2024-03-29T11:09:10.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafc37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:09:10.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:10.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:10.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafc37fe640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:09:10.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafc37fe640 1 -- 172.21.15.71:0/1989406241 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafc8004eb0 con 0x7fafdc0b4460 2024-03-29T11:09:10.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafc37fe640 1 -- 172.21.15.71:0/1989406241 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3593632570 0 0) 0x7fafc8004eb0 con 0x7fafdc0b4460 2024-03-29T11:09:10.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafc37fe640 1 -- 172.21.15.71:0/1989406241 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafdc0ae3f0 con 0x7fafdc0b4460 2024-03-29T11:09:10.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafc37fe640 1 -- 172.21.15.71:0/1989406241 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fafcc001550 con 0x7fafdc0b4460 2024-03-29T11:09:10.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafc37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:10.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafc37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:09:10.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafc37fe640 10 monclient(hunting): dump: 2024-03-29T11:09:10.939 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:10.939 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:10.940 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:10.940 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:10.940 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:10.940 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:10.940 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:10.940 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:10.940 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:10.940 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:10.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafc37fe640 1 -- 172.21.15.71:0/1989406241 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3700011929 0 0) 0x7fafdc0ae3f0 con 0x7fafdc0b4460 2024-03-29T11:09:10.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafc37fe640 1 -- 172.21.15.71:0/1989406241 >> v1:172.21.15.71:6790/0 conn(0x7fafdc0a9e00 legacy=0x7fafdc0708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:10.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafc37fe640 1 -- 172.21.15.71:0/1989406241 >> v1:172.21.15.137:6789/0 conn(0x7fafdc13f900 legacy=0x7fafdc1e6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:10.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafc37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:09:10.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafc37fe640 1 monclient: found mon.a 2024-03-29T11:09:10.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:10.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafc37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:09:10.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafc37fe640 1 -- 172.21.15.71:0/1989406241 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafdc0ad5d0 con 0x7fafdc0b4460 2024-03-29T11:09:10.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafc37fe640 10 monclient: _finish_auth 0 2024-03-29T11:09:10.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafc37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:09:10.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:10.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafdbfff640 5 monclient: authenticate success, global_id 4551 2024-03-29T11:09:10.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafdbfff640 10 monclient: _renew_subs 2024-03-29T11:09:10.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafdbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:09:10.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafdbfff640 1 -- 172.21.15.71:0/1989406241 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafdc0ac2e0 con 0x7fafdc0b4460 2024-03-29T11:09:10.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafdbfff640 10 client.4551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:10.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafdbfff640 10 monclient: _renew_subs 2024-03-29T11:09:10.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafdbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:09:10.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafdbfff640 1 -- 172.21.15.71:0/1989406241 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fafdc0ad040 con 0x7fafdc0b4460 2024-03-29T11:09:10.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.953+0000 7fafdbfff640 1 librados: init done 2024-03-29T11:09:10.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.954+0000 7fafc37fe640 1 -- 172.21.15.71:0/1989406241 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafcc0016b0 con 0x7fafdc0b4460 2024-03-29T11:09:10.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.954+0000 7fafc37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:10.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.954+0000 7fafc37fe640 1 -- 172.21.15.71:0/1989406241 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fafcc004d90 con 0x7fafdc0b4460 2024-03-29T11:09:10.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.954+0000 7fafc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:10.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.954+0000 7fafc37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:09:10.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.954+0000 7fafc37fe640 10 monclient: dump: 2024-03-29T11:09:10.942 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:10.942 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:10.942 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:10.942 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:10.942 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:10.943 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:10.943 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:10.943 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:10.943 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:10.943 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:10.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.955+0000 7fafd97fa640 4 set_mon_vals no callback set 2024-03-29T11:09:10.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.956+0000 7fafc37fe640 1 -- 172.21.15.71:0/1989406241 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7fafcc01ad60 con 0x7fafdc0b4460 2024-03-29T11:09:10.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.956+0000 7fafc37fe640 1 -- 172.21.15.71:0/1989406241 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7fafcc084b40 con 0x7fafdc0b4460 2024-03-29T11:09:10.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.956+0000 7fafc37fe640 10 client.4551.objecter ms_dispatch 0x7fafdc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:09:10.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.957+0000 7fafc37fe640 3 client.4551.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:09:10.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.957+0000 7fafdaffd640 10 monclient: get_auth_request con 0x7fafc806de30 auth_method 0 2024-03-29T11:09:10.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.957+0000 7fafc37fe640 3 client.4551.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:09:10.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.957+0000 7fafc37fe640 20 client.4551.objecter dump_active .. 0 homeless 2024-03-29T11:09:10.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.957+0000 7fafc37fe640 10 client.4551.objecter ms_handle_connect 0x7fafc806de30 2024-03-29T11:09:10.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.958+0000 7fafdbfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:10.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.958+0000 7fafdbfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:10.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.958+0000 7fafdbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:09:10.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.958+0000 7fafdbfff640 1 -- 172.21.15.71:0/1989406241 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafbc005080 con 0x7fafdc0b4460 2024-03-29T11:09:10.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.967+0000 7fafc37fe640 1 -- 172.21.15.71:0/1989406241 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fafcc050420 con 0x7fafdc0b4460 2024-03-29T11:09:10.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.967+0000 7fafc37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:10.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:10.967+0000 7fafc37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:11.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:11.293+0000 7fafdbfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:11.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:11.293+0000 7fafdbfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:11.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:11.293+0000 7fafdbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:09:11.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:11.293+0000 7fafdbfff640 1 -- 172.21.15.71:0/1989406241 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fafbc005330 con 0x7fafdc0b4460 2024-03-29T11:09:11.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:11.295+0000 7fafc37fe640 1 -- 172.21.15.71:0/1989406241 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7fafcc052350 con 0x7fafdc0b4460 2024-03-29T11:09:11.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:11.295+0000 7fafc37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:11.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:11.295+0000 7fafc37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:11.274 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:09:11.274 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:09:11.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:11.298+0000 7fafdbfff640 10 librados: watch_flush enter 2024-03-29T11:09:11.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:11.299+0000 7fafdbfff640 10 librados: watch_flush exit 2024-03-29T11:09:11.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:11.299+0000 7fafdbfff640 20 client.4551.objecter shutdown clearing up homeless session... 2024-03-29T11:09:11.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:11.299+0000 7fafdbfff640 10 client.4551.objecter successfully canceled tick 2024-03-29T11:09:11.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:11.299+0000 7fafdbfff640 1 -- 172.21.15.71:0/1989406241 >> v1:172.21.15.137:6816/973332665 conn(0x7fafc806de30 legacy=0x7fafc80702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:11.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:11.299+0000 7fafdbfff640 10 monclient: shutdown 2024-03-29T11:09:11.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:11.299+0000 7fafdbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:11.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:11.299+0000 7fafdbfff640 1 -- 172.21.15.71:0/1989406241 >> v1:172.21.15.71:6789/0 conn(0x7fafdc0b4460 legacy=0x7fafdc070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:11.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:11.301+0000 7fafdbfff640 1 -- 172.21.15.71:0/1989406241 shutdown_connections 2024-03-29T11:09:11.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:11.301+0000 7fafdbfff640 1 -- 172.21.15.71:0/1989406241 >> 172.21.15.71:0/1989406241 conn(0x7fafdc06c980 msgr2=0x7fafdc0b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:11.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:11.301+0000 7fafdbfff640 1 -- 172.21.15.71:0/1989406241 shutdown_connections 2024-03-29T11:09:11.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:11.301+0000 7fafdbfff640 1 -- 172.21.15.71:0/1989406241 wait complete. 2024-03-29T11:09:11.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:11.301+0000 7fafdbfff640 1 librados: shutdown 2024-03-29T11:09:11.297 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:09:18.104 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:09:18.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.271+0000 7f07d33cf640 10 monclient: get_monmap_and_config 2024-03-29T11:09:18.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.271+0000 7f07d33cf640 10 monclient: build_initial_monmap 2024-03-29T11:09:18.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.271+0000 7f07d33cf640 10 monclient: monmap: 2024-03-29T11:09:18.250 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:09:18.250 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:18.250 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:09:18.272669+0000 2024-03-29T11:09:18.250 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:09:18.272669+0000 2024-03-29T11:09:18.250 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:18.251 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:18.251 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:09:18.251 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:09:18.251 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:09:18.251 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:18.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.272+0000 7f07d33cf640 1 Processor -- start 2024-03-29T11:09:18.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.273+0000 7f07d33cf640 1 -- start start 2024-03-29T11:09:18.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.273+0000 7f07d33cf640 10 monclient: init 2024-03-29T11:09:18.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.274+0000 7f07d33cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:18.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.274+0000 7f07d33cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:09:18.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.274+0000 7f07d33cf640 10 monclient(hunting): picked mon.noname-b con 0x7f07cc1457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:09:18.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.274+0000 7f07d33cf640 10 monclient(hunting): picked mon.noname-c con 0x7f07cc13f650 addr v1:172.21.15.137:6789/0 2024-03-29T11:09:18.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.274+0000 7f07d33cf640 10 monclient(hunting): picked mon.noname-a con 0x7f07cc0a9d70 addr v1:172.21.15.71:6789/0 2024-03-29T11:09:18.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.274+0000 7f07d33cf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07cc0aaef0 con 0x7f07cc0a9d70 2024-03-29T11:09:18.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.274+0000 7f07d33cf640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07cc145bd0 con 0x7f07cc13f650 2024-03-29T11:09:18.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.274+0000 7f07d33cf640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07cc145eb0 con 0x7f07cc1457f0 2024-03-29T11:09:18.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.274+0000 7f07d33cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:18.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.274+0000 7f07d33cf640 10 monclient(hunting): authenticate will time out at 1314.532246s 2024-03-29T11:09:18.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.274+0000 7f07d096a640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f07cc13f650 0x7f07cc0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:49984/0 (socket says 172.21.15.71:49984) 2024-03-29T11:09:18.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.274+0000 7f07d096a640 1 -- 172.21.15.71:0/1864713629 learned_addr learned my addr 172.21.15.71:0/1864713629 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:09:18.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.274+0000 7f07c3fff640 1 -- 172.21.15.71:0/1864713629 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2830671257 0 0) 0x7f07cc145bd0 con 0x7f07cc13f650 2024-03-29T11:09:18.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.275+0000 7f07c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:09:18.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.275+0000 7f07c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:18.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.275+0000 7f07c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:18.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.275+0000 7f07c3fff640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:09:18.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.275+0000 7f07c3fff640 1 -- 172.21.15.71:0/1864713629 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07b8003520 con 0x7f07cc13f650 2024-03-29T11:09:18.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.275+0000 7f07c3fff640 1 -- 172.21.15.71:0/1864713629 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 363261622 0 0) 0x7f07cc145eb0 con 0x7f07cc1457f0 2024-03-29T11:09:18.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.275+0000 7f07c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:09:18.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.275+0000 7f07c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:18.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.275+0000 7f07c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:18.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.275+0000 7f07c3fff640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:09:18.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.275+0000 7f07c3fff640 1 -- 172.21.15.71:0/1864713629 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07cc145bd0 con 0x7f07cc1457f0 2024-03-29T11:09:18.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.275+0000 7f07c3fff640 1 -- 172.21.15.71:0/1864713629 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1788056766 0 0) 0x7f07cc145bd0 con 0x7f07cc1457f0 2024-03-29T11:09:18.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.275+0000 7f07c3fff640 1 -- 172.21.15.71:0/1864713629 >> v1:172.21.15.137:6789/0 conn(0x7f07cc13f650 legacy=0x7f07cc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:18.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.275+0000 7f07c3fff640 1 -- 172.21.15.71:0/1864713629 >> v1:172.21.15.71:6789/0 conn(0x7f07cc0a9d70 legacy=0x7f07cc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:18.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.275+0000 7f07c3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:09:18.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.275+0000 7f07c3fff640 1 monclient: found mon.noname-b 2024-03-29T11:09:18.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.275+0000 7f07c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:18.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.275+0000 7f07c3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:09:18.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.275+0000 7f07c3fff640 1 -- 172.21.15.71:0/1864713629 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07cc146340 con 0x7f07cc1457f0 2024-03-29T11:09:18.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.275+0000 7f07c3fff640 10 monclient: _finish_auth 0 2024-03-29T11:09:18.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.275+0000 7f07c3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:09:18.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.275+0000 7f07c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:18.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.275+0000 7f07c3fff640 1 -- 172.21.15.71:0/1864713629 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f07c8001670 con 0x7f07cc1457f0 2024-03-29T11:09:18.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.275+0000 7f07c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:18.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.275+0000 7f07c3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:09:18.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.275+0000 7f07c3fff640 10 monclient: dump: 2024-03-29T11:09:18.257 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:18.257 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:18.257 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:18.257 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:18.257 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:18.258 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:18.258 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:18.258 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:18.258 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:18.258 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:18.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.276+0000 7f07d33cf640 5 monclient: authenticate success, global_id 4607 2024-03-29T11:09:18.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.276+0000 7f07d33cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:18.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.276+0000 7f07c3fff640 1 -- 172.21.15.71:0/1864713629 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07c80017d0 con 0x7f07cc1457f0 2024-03-29T11:09:18.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.276+0000 7f07c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:18.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.276+0000 7f07c3fff640 1 -- 172.21.15.71:0/1864713629 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f07c8001ba0 con 0x7f07cc1457f0 2024-03-29T11:09:18.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.276+0000 7f07c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:18.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.276+0000 7f07c3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:18.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.276+0000 7f07c3fff640 10 monclient: dump: 2024-03-29T11:09:18.259 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:18.259 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:18.259 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:18.259 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:18.259 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:18.259 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:18.259 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:18.259 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:18.259 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:18.259 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:18.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.276+0000 7f07d33cf640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:18.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.276+0000 7f07d33cf640 4 set_mon_vals no callback set 2024-03-29T11:09:18.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.277+0000 7f07d33cf640 10 monclient: shutdown 2024-03-29T11:09:18.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.277+0000 7f07d33cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:18.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.277+0000 7f07d33cf640 1 -- 172.21.15.71:0/1864713629 >> v1:172.21.15.71:6790/0 conn(0x7f07cc1457f0 legacy=0x7f07cc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:18.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.277+0000 7f07d33cf640 1 -- 172.21.15.71:0/1864713629 shutdown_connections 2024-03-29T11:09:18.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.277+0000 7f07d33cf640 1 -- 172.21.15.71:0/1864713629 >> 172.21.15.71:0/1864713629 conn(0x7f07cc066a00 msgr2=0x7f07cc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:18.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.278+0000 7f07d33cf640 1 -- 172.21.15.71:0/1864713629 shutdown_connections 2024-03-29T11:09:18.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.278+0000 7f07d33cf640 1 -- 172.21.15.71:0/1864713629 wait complete. 2024-03-29T11:09:18.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.278+0000 7f07d33cf640 10 monclient: build_initial_monmap 2024-03-29T11:09:18.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.278+0000 7f07d33cf640 10 monclient: monmap: 2024-03-29T11:09:18.268 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:09:18.268 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:18.268 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:09:18.268 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:09:18.269 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:18.269 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:18.269 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:09:18.269 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:09:18.269 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:09:18.269 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:18.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.279+0000 7f07d33cf640 1 librados: starting msgr at 2024-03-29T11:09:18.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.279+0000 7f07d33cf640 1 librados: starting objecter 2024-03-29T11:09:18.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.279+0000 7f07d33cf640 1 Processor -- start 2024-03-29T11:09:18.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.279+0000 7f07d33cf640 1 -- start start 2024-03-29T11:09:18.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.279+0000 7f07d33cf640 1 librados: setting wanted keys 2024-03-29T11:09:18.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.279+0000 7f07d33cf640 1 librados: calling monclient init 2024-03-29T11:09:18.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.279+0000 7f07d33cf640 10 monclient: init 2024-03-29T11:09:18.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.280+0000 7f07d33cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:18.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.280+0000 7f07d33cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:09:18.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.280+0000 7f07d33cf640 10 monclient(hunting): picked mon.noname-a con 0x7f07cc0a9d70 addr v1:172.21.15.71:6789/0 2024-03-29T11:09:18.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.280+0000 7f07d33cf640 10 monclient(hunting): picked mon.noname-b con 0x7f07cc13f650 addr v1:172.21.15.137:6789/0 2024-03-29T11:09:18.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.280+0000 7f07d33cf640 10 monclient(hunting): picked mon.noname-c con 0x7f07cc1457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:09:18.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.280+0000 7f07d116b640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f07cc0a9d70 0x7f07cc1d76f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:49486/0 (socket says 172.21.15.71:49486) 2024-03-29T11:09:18.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.280+0000 7f07d116b640 1 -- 172.21.15.71:0/1854438341 learned_addr learned my addr 172.21.15.71:0/1854438341 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:09:18.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.280+0000 7f07d33cf640 1 -- 172.21.15.71:0/1854438341 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07cc0af530 con 0x7f07cc0a9d70 2024-03-29T11:09:18.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.280+0000 7f07c1ffb640 10 client.?.objecter ms_handle_connect 0x7f07cc0a9d70 2024-03-29T11:09:18.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.280+0000 7f07c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:18.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.281+0000 7f07c1ffb640 10 client.?.objecter ms_handle_connect 0x7f07cc13f650 2024-03-29T11:09:18.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.281+0000 7f07c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:18.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.281+0000 7f07c1ffb640 10 client.?.objecter ms_handle_connect 0x7f07cc1457f0 2024-03-29T11:09:18.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.281+0000 7f07c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:18.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.281+0000 7f07d33cf640 1 -- 172.21.15.71:0/1854438341 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07cc1d7e00 con 0x7f07cc13f650 2024-03-29T11:09:18.271 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.281+0000 7f07d33cf640 1 -- 172.21.15.71:0/1854438341 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07cc1d7f90 con 0x7f07cc1457f0 2024-03-29T11:09:18.271 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.281+0000 7f07d33cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:18.271 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.281+0000 7f07d33cf640 10 monclient(hunting): authenticate will time out at 1314.539676s 2024-03-29T11:09:18.271 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.281+0000 7f07c1ffb640 1 -- 172.21.15.71:0/1854438341 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370731513 0 0) 0x7f07cc1d7e00 con 0x7f07cc13f650 2024-03-29T11:09:18.271 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.281+0000 7f07c1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:09:18.271 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.281+0000 7f07c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:18.271 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.282+0000 7f07c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:18.271 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.282+0000 7f07c1ffb640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:09:18.271 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.282+0000 7f07c1ffb640 1 -- 172.21.15.71:0/1854438341 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07b0004eb0 con 0x7f07cc13f650 2024-03-29T11:09:18.271 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.282+0000 7f07c1ffb640 1 -- 172.21.15.71:0/1854438341 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581139325 0 0) 0x7f07cc1d7f90 con 0x7f07cc1457f0 2024-03-29T11:09:18.271 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.282+0000 7f07c1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:09:18.271 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.282+0000 7f07c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:18.271 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.282+0000 7f07c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:18.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.282+0000 7f07c1ffb640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:09:18.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.282+0000 7f07c1ffb640 1 -- 172.21.15.71:0/1854438341 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07cc1d7e00 con 0x7f07cc1457f0 2024-03-29T11:09:18.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.282+0000 7f07c1ffb640 1 -- 172.21.15.71:0/1854438341 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320552083 0 0) 0x7f07cc0af530 con 0x7f07cc0a9d70 2024-03-29T11:09:18.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.282+0000 7f07c1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:09:18.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.282+0000 7f07c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:18.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.282+0000 7f07c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:18.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.282+0000 7f07c1ffb640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:09:18.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.282+0000 7f07c1ffb640 1 -- 172.21.15.71:0/1854438341 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07cc1d7f90 con 0x7f07cc0a9d70 2024-03-29T11:09:18.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.282+0000 7f07c1ffb640 1 -- 172.21.15.71:0/1854438341 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2905139341 0 0) 0x7f07cc1d7e00 con 0x7f07cc1457f0 2024-03-29T11:09:18.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.283+0000 7f07c1ffb640 1 -- 172.21.15.71:0/1854438341 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07cc0af530 con 0x7f07cc1457f0 2024-03-29T11:09:18.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.283+0000 7f07c1ffb640 1 -- 172.21.15.71:0/1854438341 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f07c4001bb0 con 0x7f07cc1457f0 2024-03-29T11:09:18.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.283+0000 7f07c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:18.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.283+0000 7f07c1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:09:18.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.283+0000 7f07c1ffb640 10 monclient(hunting): dump: 2024-03-29T11:09:18.272 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:18.272 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:18.273 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:18.273 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:18.273 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:18.273 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:18.273 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:18.273 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:18.273 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:18.273 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:18.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.283+0000 7f07c1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:18.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.283+0000 7f07c1ffb640 1 -- 172.21.15.71:0/1854438341 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1906312412 0 0) 0x7f07cc1d7f90 con 0x7f07cc0a9d70 2024-03-29T11:09:18.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.283+0000 7f07c1ffb640 1 -- 172.21.15.71:0/1854438341 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07cc1d7e00 con 0x7f07cc0a9d70 2024-03-29T11:09:18.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.283+0000 7f07c1ffb640 1 -- 172.21.15.71:0/1854438341 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1905414633 0 0) 0x7f07cc0af530 con 0x7f07cc1457f0 2024-03-29T11:09:18.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.283+0000 7f07c1ffb640 1 -- 172.21.15.71:0/1854438341 >> v1:172.21.15.137:6789/0 conn(0x7f07cc13f650 legacy=0x7f07cc139d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:18.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.283+0000 7f07c1ffb640 1 -- 172.21.15.71:0/1854438341 >> v1:172.21.15.71:6789/0 conn(0x7f07cc0a9d70 legacy=0x7f07cc1d76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:18.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.283+0000 7f07c1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:09:18.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.283+0000 7f07c1ffb640 1 monclient: found mon.c 2024-03-29T11:09:18.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.284+0000 7f07c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:18.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.284+0000 7f07c1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:18.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.284+0000 7f07c1ffb640 1 -- 172.21.15.71:0/1854438341 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07cc1420a0 con 0x7f07cc1457f0 2024-03-29T11:09:18.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.284+0000 7f07c1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:09:18.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.284+0000 7f07c1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:09:18.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.284+0000 7f07c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:18.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.284+0000 7f07c1ffb640 1 -- 172.21.15.71:0/1854438341 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07c4003970 con 0x7f07cc1457f0 2024-03-29T11:09:18.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.284+0000 7f07d33cf640 5 monclient: authenticate success, global_id 4610 2024-03-29T11:09:18.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.284+0000 7f07d33cf640 10 monclient: _renew_subs 2024-03-29T11:09:18.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.284+0000 7f07d33cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:18.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.284+0000 7f07d33cf640 1 -- 172.21.15.71:0/1854438341 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07cc1422f0 con 0x7f07cc1457f0 2024-03-29T11:09:18.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.284+0000 7f07c1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:18.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.284+0000 7f07c1ffb640 1 -- 172.21.15.71:0/1854438341 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f07c4003d00 con 0x7f07cc1457f0 2024-03-29T11:09:18.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.284+0000 7f07c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:18.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.284+0000 7f07c1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:18.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.284+0000 7f07c1ffb640 10 monclient: dump: 2024-03-29T11:09:18.275 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:18.275 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:18.275 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:18.275 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:18.275 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:18.275 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:18.275 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:18.276 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:18.276 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:18.276 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:18.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.284+0000 7f07d33cf640 10 client.4610.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:18.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.284+0000 7f07d33cf640 10 monclient: _renew_subs 2024-03-29T11:09:18.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.284+0000 7f07d33cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:18.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.284+0000 7f07d33cf640 1 -- 172.21.15.71:0/1854438341 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f07cc13a650 con 0x7f07cc1457f0 2024-03-29T11:09:18.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.284+0000 7f07c37fe640 4 set_mon_vals no callback set 2024-03-29T11:09:18.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.284+0000 7f07d33cf640 1 librados: init done 2024-03-29T11:09:18.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.286+0000 7f07d33cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:18.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.286+0000 7f07d33cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:18.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.286+0000 7f07d33cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:18.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.286+0000 7f07d33cf640 1 -- 172.21.15.71:0/1854438341 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07cc075f10 con 0x7f07cc1457f0 2024-03-29T11:09:18.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.287+0000 7f07c1ffb640 1 -- 172.21.15.71:0/1854438341 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f07c4003e60 con 0x7f07cc1457f0 2024-03-29T11:09:18.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.287+0000 7f07d116b640 10 monclient: get_auth_request con 0x7f07b006df10 auth_method 0 2024-03-29T11:09:18.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.287+0000 7f07c1ffb640 1 -- 172.21.15.71:0/1854438341 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f07c4083a80 con 0x7f07cc1457f0 2024-03-29T11:09:18.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.287+0000 7f07c1ffb640 10 client.4610.objecter ms_dispatch 0x7f07cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:09:18.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.287+0000 7f07c1ffb640 3 client.4610.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:09:18.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.287+0000 7f07c1ffb640 3 client.4610.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:09:18.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.288+0000 7f07c1ffb640 20 client.4610.objecter dump_active .. 0 homeless 2024-03-29T11:09:18.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.288+0000 7f07c1ffb640 10 client.4610.objecter ms_handle_connect 0x7f07b006df10 2024-03-29T11:09:18.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.295+0000 7f07c1ffb640 1 -- 172.21.15.71:0/1854438341 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f07c404f360 con 0x7f07cc1457f0 2024-03-29T11:09:18.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.295+0000 7f07c1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:18.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.295+0000 7f07c1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:18.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.633+0000 7f07d33cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:18.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.633+0000 7f07d33cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:18.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.633+0000 7f07d33cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:18.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.633+0000 7f07d33cf640 1 -- 172.21.15.71:0/1854438341 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07cc0b3680 con 0x7f07cc1457f0 2024-03-29T11:09:18.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.634+0000 7f07c1ffb640 1 -- 172.21.15.71:0/1854438341 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f07c4051290 con 0x7f07cc1457f0 2024-03-29T11:09:18.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.634+0000 7f07c1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:18.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.634+0000 7f07c1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:18.613 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:09:18.613 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:09:18.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.637+0000 7f07d33cf640 10 librados: watch_flush enter 2024-03-29T11:09:18.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.637+0000 7f07d33cf640 10 librados: watch_flush exit 2024-03-29T11:09:18.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.637+0000 7f07d33cf640 20 client.4610.objecter shutdown clearing up homeless session... 2024-03-29T11:09:18.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.637+0000 7f07d33cf640 10 client.4610.objecter successfully canceled tick 2024-03-29T11:09:18.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.638+0000 7f07d33cf640 1 -- 172.21.15.71:0/1854438341 >> v1:172.21.15.137:6816/973332665 conn(0x7f07b006df10 legacy=0x7f07b00703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:18.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.638+0000 7f07d33cf640 10 monclient: shutdown 2024-03-29T11:09:18.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.638+0000 7f07d33cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:18.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.638+0000 7f07d33cf640 1 -- 172.21.15.71:0/1854438341 >> v1:172.21.15.71:6790/0 conn(0x7f07cc1457f0 legacy=0x7f07cc0ab9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:18.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.638+0000 7f07d33cf640 1 -- 172.21.15.71:0/1854438341 shutdown_connections 2024-03-29T11:09:18.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.638+0000 7f07d33cf640 1 -- 172.21.15.71:0/1854438341 >> 172.21.15.71:0/1854438341 conn(0x7f07cc066a00 msgr2=0x7f07cc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:18.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.638+0000 7f07d33cf640 1 -- 172.21.15.71:0/1854438341 shutdown_connections 2024-03-29T11:09:18.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.638+0000 7f07d33cf640 1 -- 172.21.15.71:0/1854438341 wait complete. 2024-03-29T11:09:18.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:18.638+0000 7f07d33cf640 1 librados: shutdown 2024-03-29T11:09:18.630 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:09:25.441 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:09:25.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.599+0000 7fbbfc225640 10 monclient: get_monmap_and_config 2024-03-29T11:09:25.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.599+0000 7fbbfc225640 10 monclient: build_initial_monmap 2024-03-29T11:09:25.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.599+0000 7fbbfc225640 10 monclient: monmap: 2024-03-29T11:09:25.578 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:09:25.578 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:25.578 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:09:25.600494+0000 2024-03-29T11:09:25.578 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:09:25.600494+0000 2024-03-29T11:09:25.578 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:25.578 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:25.578 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:09:25.578 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:09:25.579 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:09:25.579 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:25.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.600+0000 7fbbfc225640 1 Processor -- start 2024-03-29T11:09:25.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.600+0000 7fbbfc225640 1 -- start start 2024-03-29T11:09:25.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.600+0000 7fbbfc225640 10 monclient: init 2024-03-29T11:09:25.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.601+0000 7fbbfc225640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:25.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.601+0000 7fbbfc225640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:09:25.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.601+0000 7fbbfc225640 10 monclient(hunting): picked mon.noname-a con 0x7fbbf40a9750 addr v1:172.21.15.71:6789/0 2024-03-29T11:09:25.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.601+0000 7fbbf9fc1640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fbbf40a9750 0x7fbbf413dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:41354/0 (socket says 172.21.15.71:41354) 2024-03-29T11:09:25.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.601+0000 7fbbf9fc1640 1 -- 172.21.15.71:0/3610059084 learned_addr learned my addr 172.21.15.71:0/3610059084 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:09:25.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.601+0000 7fbbfc225640 10 monclient(hunting): picked mon.noname-b con 0x7fbbf413e1e0 addr v1:172.21.15.71:6790/0 2024-03-29T11:09:25.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.601+0000 7fbbfc225640 10 monclient(hunting): picked mon.noname-c con 0x7fbbf4071c30 addr v1:172.21.15.137:6789/0 2024-03-29T11:09:25.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.602+0000 7fbbfc225640 1 -- 172.21.15.71:0/3610059084 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf40ab300 con 0x7fbbf40a9750 2024-03-29T11:09:25.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.602+0000 7fbbfc225640 1 -- 172.21.15.71:0/3610059084 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf40ac2f0 con 0x7fbbf4071c30 2024-03-29T11:09:25.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.602+0000 7fbbfc225640 1 -- 172.21.15.71:0/3610059084 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf40ad8d0 con 0x7fbbf413e1e0 2024-03-29T11:09:25.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.602+0000 7fbbfc225640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:25.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.602+0000 7fbbfc225640 10 monclient(hunting): authenticate will time out at 1321.860260s 2024-03-29T11:09:25.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.602+0000 7fbbf8fbf640 1 -- 172.21.15.71:0/3610059084 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4053393301 0 0) 0x7fbbf40ab300 con 0x7fbbf40a9750 2024-03-29T11:09:25.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.602+0000 7fbbf8fbf640 10 monclient(hunting): _negotiate 2024-03-29T11:09:25.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.602+0000 7fbbf8fbf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:25.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.602+0000 7fbbf8fbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:25.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.602+0000 7fbbf8fbf640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:09:25.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.602+0000 7fbbf8fbf640 1 -- 172.21.15.71:0/3610059084 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbe0003440 con 0x7fbbf40a9750 2024-03-29T11:09:25.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.602+0000 7fbbf8fbf640 1 -- 172.21.15.71:0/3610059084 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 319712760 0 0) 0x7fbbe0003440 con 0x7fbbf40a9750 2024-03-29T11:09:25.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.602+0000 7fbbf8fbf640 1 -- 172.21.15.71:0/3610059084 >> v1:172.21.15.71:6790/0 conn(0x7fbbf413e1e0 legacy=0x7fbbf40ac710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:09:25.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.605+0000 7fbbf8fbf640 1 -- 172.21.15.71:0/3610059084 >> v1:172.21.15.137:6789/0 conn(0x7fbbf4071c30 legacy=0x7fbbf40ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:25.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.605+0000 7fbbf8fbf640 10 monclient: _finish_hunting 0 2024-03-29T11:09:25.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.605+0000 7fbbf8fbf640 1 monclient: found mon.noname-a 2024-03-29T11:09:25.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.605+0000 7fbbf8fbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:25.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.605+0000 7fbbf8fbf640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:09:25.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.605+0000 7fbbf8fbf640 1 -- 172.21.15.71:0/3610059084 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf40ada60 con 0x7fbbf40a9750 2024-03-29T11:09:25.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.605+0000 7fbbf8fbf640 10 monclient: _finish_auth 0 2024-03-29T11:09:25.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.605+0000 7fbbf8fbf640 10 monclient: _check_auth_tickets 2024-03-29T11:09:25.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.605+0000 7fbbf8fbf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:25.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.605+0000 7fbbf8fbf640 1 -- 172.21.15.71:0/3610059084 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fbbf0001690 con 0x7fbbf40a9750 2024-03-29T11:09:25.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.605+0000 7fbbf8fbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:25.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.605+0000 7fbbf8fbf640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:09:25.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.605+0000 7fbbf8fbf640 10 monclient: dump: 2024-03-29T11:09:25.585 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:25.585 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:25.585 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:25.585 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:25.585 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:25.585 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:25.585 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:25.585 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:25.585 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:25.585 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:25.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.605+0000 7fbbfc225640 5 monclient: authenticate success, global_id 4560 2024-03-29T11:09:25.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.605+0000 7fbbfc225640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:25.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.605+0000 7fbbf8fbf640 1 -- 172.21.15.71:0/3610059084 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbf0001860 con 0x7fbbf40a9750 2024-03-29T11:09:25.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.605+0000 7fbbf8fbf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:25.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.605+0000 7fbbf8fbf640 1 -- 172.21.15.71:0/3610059084 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fbbf0001c70 con 0x7fbbf40a9750 2024-03-29T11:09:25.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.605+0000 7fbbf8fbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:25.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.605+0000 7fbbf8fbf640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:09:25.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.605+0000 7fbbf8fbf640 10 monclient: dump: 2024-03-29T11:09:25.586 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:25.586 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:25.586 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:25.586 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:25.586 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:25.586 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:25.587 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:25.587 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:25.587 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:25.587 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:25.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.605+0000 7fbbfc225640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:25.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.605+0000 7fbbfc225640 4 set_mon_vals no callback set 2024-03-29T11:09:25.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.606+0000 7fbbfc225640 10 monclient: shutdown 2024-03-29T11:09:25.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.606+0000 7fbbfc225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:25.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.606+0000 7fbbfc225640 1 -- 172.21.15.71:0/3610059084 >> v1:172.21.15.71:6789/0 conn(0x7fbbf40a9750 legacy=0x7fbbf413dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:25.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.607+0000 7fbbfc225640 1 -- 172.21.15.71:0/3610059084 shutdown_connections 2024-03-29T11:09:25.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.607+0000 7fbbfc225640 1 -- 172.21.15.71:0/3610059084 >> 172.21.15.71:0/3610059084 conn(0x7fbbf406b660 msgr2=0x7fbbf406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:25.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.607+0000 7fbbfc225640 1 -- 172.21.15.71:0/3610059084 shutdown_connections 2024-03-29T11:09:25.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.607+0000 7fbbfc225640 1 -- 172.21.15.71:0/3610059084 wait complete. 2024-03-29T11:09:25.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.608+0000 7fbbfc225640 10 monclient: build_initial_monmap 2024-03-29T11:09:25.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.608+0000 7fbbfc225640 10 monclient: monmap: 2024-03-29T11:09:25.595 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:09:25.595 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:25.595 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:09:25.595 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:09:25.595 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:25.595 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:25.595 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:09:25.595 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:09:25.595 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:09:25.595 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:25.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.608+0000 7fbbfc225640 1 librados: starting msgr at 2024-03-29T11:09:25.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.608+0000 7fbbfc225640 1 librados: starting objecter 2024-03-29T11:09:25.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.608+0000 7fbbfc225640 1 Processor -- start 2024-03-29T11:09:25.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.608+0000 7fbbfc225640 1 -- start start 2024-03-29T11:09:25.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.608+0000 7fbbfc225640 1 librados: setting wanted keys 2024-03-29T11:09:25.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.608+0000 7fbbfc225640 1 librados: calling monclient init 2024-03-29T11:09:25.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.608+0000 7fbbfc225640 10 monclient: init 2024-03-29T11:09:25.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.609+0000 7fbbfc225640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:25.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.609+0000 7fbbfc225640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:09:25.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.609+0000 7fbbfc225640 10 monclient(hunting): picked mon.noname-a con 0x7fbbf4071c30 addr v1:172.21.15.71:6789/0 2024-03-29T11:09:25.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.609+0000 7fbbfc225640 10 monclient(hunting): picked mon.noname-c con 0x7fbbf40a9750 addr v1:172.21.15.71:6790/0 2024-03-29T11:09:25.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.609+0000 7fbbfc225640 10 monclient(hunting): picked mon.noname-b con 0x7fbbf413e1e0 addr v1:172.21.15.137:6789/0 2024-03-29T11:09:25.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.609+0000 7fbbfc225640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf4141460 con 0x7fbbf4071c30 2024-03-29T11:09:25.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.609+0000 7fbbfc225640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf4142390 con 0x7fbbf413e1e0 2024-03-29T11:09:25.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.609+0000 7fbbfc225640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf41416d0 con 0x7fbbf40a9750 2024-03-29T11:09:25.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.609+0000 7fbbfc225640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:25.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.609+0000 7fbbfc225640 10 monclient(hunting): authenticate will time out at 1321.867815s 2024-03-29T11:09:25.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.609+0000 7fbbf9fc1640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fbbf4071c30 0x7fbbf413fbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:44062/0 (socket says 172.21.15.71:44062) 2024-03-29T11:09:25.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.609+0000 7fbbf9fc1640 1 -- 172.21.15.71:0/3616747517 learned_addr learned my addr 172.21.15.71:0/3616747517 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:09:25.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.609+0000 7fbbea7fc640 10 client.?.objecter ms_handle_connect 0x7fbbf4071c30 2024-03-29T11:09:25.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.610+0000 7fbbea7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:25.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.610+0000 7fbbea7fc640 1 -- 172.21.15.71:0/3616747517 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692516516 0 0) 0x7fbbf4141460 con 0x7fbbf4071c30 2024-03-29T11:09:25.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.610+0000 7fbbea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:09:25.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.610+0000 7fbbea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:25.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.610+0000 7fbbea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:25.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.610+0000 7fbbea7fc640 10 monclient(hunting): my global_id is 4563 2024-03-29T11:09:25.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.610+0000 7fbbea7fc640 1 -- 172.21.15.71:0/3616747517 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbd8004eb0 con 0x7fbbf4071c30 2024-03-29T11:09:25.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.610+0000 7fbbea7fc640 10 client.?.objecter ms_handle_connect 0x7fbbf413e1e0 2024-03-29T11:09:25.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.610+0000 7fbbea7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:25.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.610+0000 7fbbea7fc640 1 -- 172.21.15.71:0/3616747517 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2123555414 0 0) 0x7fbbd8004eb0 con 0x7fbbf4071c30 2024-03-29T11:09:25.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.610+0000 7fbbea7fc640 1 -- 172.21.15.71:0/3616747517 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbf4141460 con 0x7fbbf4071c30 2024-03-29T11:09:25.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.610+0000 7fbbea7fc640 1 -- 172.21.15.71:0/3616747517 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fbbf0001b70 con 0x7fbbf4071c30 2024-03-29T11:09:25.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.610+0000 7fbbea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:25.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.610+0000 7fbbea7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:09:25.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.610+0000 7fbbea7fc640 10 monclient(hunting): dump: 2024-03-29T11:09:25.598 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:25.598 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:25.598 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:25.598 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:25.598 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:25.598 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:25.598 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:25.598 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:25.599 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:25.599 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:25.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.610+0000 7fbbea7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:25.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.610+0000 7fbbea7fc640 1 -- 172.21.15.71:0/3616747517 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1607407713 0 0) 0x7fbbf4141460 con 0x7fbbf4071c30 2024-03-29T11:09:25.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.610+0000 7fbbea7fc640 1 -- 172.21.15.71:0/3616747517 >> v1:172.21.15.71:6790/0 conn(0x7fbbf40a9750 legacy=0x7fbbf4136d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:09:25.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.613+0000 7fbbea7fc640 1 -- 172.21.15.71:0/3616747517 >> v1:172.21.15.137:6789/0 conn(0x7fbbf413e1e0 legacy=0x7fbbf41e1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:25.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.613+0000 7fbbea7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:09:25.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.613+0000 7fbbea7fc640 1 monclient: found mon.a 2024-03-29T11:09:25.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.613+0000 7fbbea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:25.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.613+0000 7fbbea7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:09:25.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.613+0000 7fbbea7fc640 1 -- 172.21.15.71:0/3616747517 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf41419b0 con 0x7fbbf4071c30 2024-03-29T11:09:25.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.613+0000 7fbbea7fc640 10 monclient: _finish_auth 0 2024-03-29T11:09:25.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.613+0000 7fbbea7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:09:25.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.613+0000 7fbbea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:25.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.613+0000 7fbbfc225640 5 monclient: authenticate success, global_id 4563 2024-03-29T11:09:25.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.613+0000 7fbbfc225640 10 monclient: _renew_subs 2024-03-29T11:09:25.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.613+0000 7fbbfc225640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:09:25.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.613+0000 7fbbfc225640 1 -- 172.21.15.71:0/3616747517 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbf4141c90 con 0x7fbbf4071c30 2024-03-29T11:09:25.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.613+0000 7fbbfc225640 10 client.4563.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:25.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.613+0000 7fbbfc225640 10 monclient: _renew_subs 2024-03-29T11:09:25.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.613+0000 7fbbfc225640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:09:25.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.613+0000 7fbbfc225640 1 -- 172.21.15.71:0/3616747517 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf4142610 con 0x7fbbf4071c30 2024-03-29T11:09:25.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.613+0000 7fbbfc225640 1 librados: init done 2024-03-29T11:09:25.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.613+0000 7fbbea7fc640 1 -- 172.21.15.71:0/3616747517 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbf0001cd0 con 0x7fbbf4071c30 2024-03-29T11:09:25.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.613+0000 7fbbea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:25.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.613+0000 7fbbea7fc640 1 -- 172.21.15.71:0/3616747517 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fbbf0004d90 con 0x7fbbf4071c30 2024-03-29T11:09:25.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.613+0000 7fbbea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:25.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.613+0000 7fbbea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:09:25.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.613+0000 7fbbea7fc640 10 monclient: dump: 2024-03-29T11:09:25.601 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:25.601 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:25.601 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:25.601 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:25.601 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:25.601 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:25.601 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:25.601 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:25.601 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:25.601 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:25.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.613+0000 7fbbeb7fe640 4 set_mon_vals no callback set 2024-03-29T11:09:25.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.615+0000 7fbbfc225640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:25.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.615+0000 7fbbfc225640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:25.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.615+0000 7fbbfc225640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:09:25.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.615+0000 7fbbfc225640 1 -- 172.21.15.71:0/3616747517 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbdc005080 con 0x7fbbf4071c30 2024-03-29T11:09:25.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.616+0000 7fbbea7fc640 1 -- 172.21.15.71:0/3616747517 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7fbbf001ad60 con 0x7fbbf4071c30 2024-03-29T11:09:25.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.616+0000 7fbbea7fc640 1 -- 172.21.15.71:0/3616747517 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7fbbf0084d80 con 0x7fbbf4071c30 2024-03-29T11:09:25.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.616+0000 7fbbea7fc640 10 client.4563.objecter ms_dispatch 0x7fbbf4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:09:25.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.616+0000 7fbbea7fc640 3 client.4563.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:09:25.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.616+0000 7fbbea7fc640 3 client.4563.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:09:25.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.616+0000 7fbbea7fc640 20 client.4563.objecter dump_active .. 0 homeless 2024-03-29T11:09:25.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.617+0000 7fbbf97c0640 10 monclient: get_auth_request con 0x7fbbd806db00 auth_method 0 2024-03-29T11:09:25.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.617+0000 7fbbea7fc640 10 client.4563.objecter ms_handle_connect 0x7fbbd806db00 2024-03-29T11:09:25.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.623+0000 7fbbea7fc640 1 -- 172.21.15.71:0/3616747517 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbbf0050490 con 0x7fbbf4071c30 2024-03-29T11:09:25.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.623+0000 7fbbea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:25.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.623+0000 7fbbea7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:25.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.932+0000 7fbbfc225640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:25.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.932+0000 7fbbfc225640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:25.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.932+0000 7fbbfc225640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:09:25.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.932+0000 7fbbfc225640 1 -- 172.21.15.71:0/3616747517 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbbdc005330 con 0x7fbbf4071c30 2024-03-29T11:09:25.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.933+0000 7fbbea7fc640 1 -- 172.21.15.71:0/3616747517 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7fbbf0052590 con 0x7fbbf4071c30 2024-03-29T11:09:25.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.933+0000 7fbbea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:25.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.933+0000 7fbbea7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:25.912 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:09:25.912 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:09:25.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.935+0000 7fbbfc225640 10 librados: watch_flush enter 2024-03-29T11:09:25.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.936+0000 7fbbfc225640 10 librados: watch_flush exit 2024-03-29T11:09:25.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.936+0000 7fbbfc225640 20 client.4563.objecter shutdown clearing up homeless session... 2024-03-29T11:09:25.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.936+0000 7fbbfc225640 10 client.4563.objecter successfully canceled tick 2024-03-29T11:09:25.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.936+0000 7fbbfc225640 1 -- 172.21.15.71:0/3616747517 >> v1:172.21.15.137:6816/973332665 conn(0x7fbbd806db00 legacy=0x7fbbd806ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:25.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.936+0000 7fbbfc225640 10 monclient: shutdown 2024-03-29T11:09:25.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.936+0000 7fbbfc225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:25.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.936+0000 7fbbfc225640 1 -- 172.21.15.71:0/3616747517 >> v1:172.21.15.71:6789/0 conn(0x7fbbf4071c30 legacy=0x7fbbf413fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:25.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.936+0000 7fbbfc225640 1 -- 172.21.15.71:0/3616747517 shutdown_connections 2024-03-29T11:09:25.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.936+0000 7fbbfc225640 1 -- 172.21.15.71:0/3616747517 >> 172.21.15.71:0/3616747517 conn(0x7fbbf406b660 msgr2=0x7fbbf413aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:25.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.936+0000 7fbbfc225640 1 -- 172.21.15.71:0/3616747517 shutdown_connections 2024-03-29T11:09:25.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.937+0000 7fbbfc225640 1 -- 172.21.15.71:0/3616747517 wait complete. 2024-03-29T11:09:25.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:25.937+0000 7fbbfc225640 1 librados: shutdown 2024-03-29T11:09:25.928 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:09:32.737 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:09:32.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.905+0000 7f2f91a16640 10 monclient: get_monmap_and_config 2024-03-29T11:09:32.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.905+0000 7f2f91a16640 10 monclient: build_initial_monmap 2024-03-29T11:09:32.884 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.905+0000 7f2f91a16640 10 monclient: monmap: 2024-03-29T11:09:32.884 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:09:32.884 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:32.884 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:09:32.906796+0000 2024-03-29T11:09:32.884 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:09:32.906796+0000 2024-03-29T11:09:32.885 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:32.885 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:32.885 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:09:32.885 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:09:32.885 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:09:32.885 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:32.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.906+0000 7f2f91a16640 1 Processor -- start 2024-03-29T11:09:32.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.907+0000 7f2f91a16640 1 -- start start 2024-03-29T11:09:32.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.907+0000 7f2f91a16640 10 monclient: init 2024-03-29T11:09:32.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.908+0000 7f2f91a16640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:32.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.908+0000 7f2f91a16640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:09:32.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.908+0000 7f2f91a16640 10 monclient(hunting): picked mon.noname-c con 0x7f2f8c1457f0 addr v1:172.21.15.137:6789/0 2024-03-29T11:09:32.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.908+0000 7f2f91a16640 10 monclient(hunting): picked mon.noname-b con 0x7f2f8c13f650 addr v1:172.21.15.71:6790/0 2024-03-29T11:09:32.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.908+0000 7f2f8affd640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f2f8c1457f0 0x7f2f8c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:47204/0 (socket says 172.21.15.71:47204) 2024-03-29T11:09:32.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.908+0000 7f2f8a7fc640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f2f8c13f650 0x7f2f8c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:46612/0 (socket says 172.21.15.71:46612) 2024-03-29T11:09:32.890 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.908+0000 7f2f91a16640 10 monclient(hunting): picked mon.noname-a con 0x7f2f8c0a9d70 addr v1:172.21.15.71:6789/0 2024-03-29T11:09:32.890 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.908+0000 7f2f8affd640 1 -- 172.21.15.71:0/1469233997 learned_addr learned my addr 172.21.15.71:0/1469233997 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:09:32.890 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.909+0000 7f2f91a16640 1 -- 172.21.15.71:0/1469233997 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8c0aaef0 con 0x7f2f8c0a9d70 2024-03-29T11:09:32.890 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.909+0000 7f2f91a16640 1 -- 172.21.15.71:0/1469233997 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8c145bd0 con 0x7f2f8c1457f0 2024-03-29T11:09:32.890 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.909+0000 7f2f91a16640 1 -- 172.21.15.71:0/1469233997 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8c145eb0 con 0x7f2f8c13f650 2024-03-29T11:09:32.890 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.909+0000 7f2f91a16640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:32.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.909+0000 7f2f91a16640 10 monclient(hunting): authenticate will time out at 1329.167723s 2024-03-29T11:09:32.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.909+0000 7f2f89ffb640 1 -- 172.21.15.71:0/1469233997 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1428308586 0 0) 0x7f2f8c145eb0 con 0x7f2f8c13f650 2024-03-29T11:09:32.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.909+0000 7f2f89ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:09:32.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.909+0000 7f2f89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:32.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.909+0000 7f2f89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:32.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.909+0000 7f2f89ffb640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:09:32.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.909+0000 7f2f89ffb640 1 -- 172.21.15.71:0/1469233997 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f74003520 con 0x7f2f8c13f650 2024-03-29T11:09:32.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.910+0000 7f2f89ffb640 1 -- 172.21.15.71:0/1469233997 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 606854121 0 0) 0x7f2f74003520 con 0x7f2f8c13f650 2024-03-29T11:09:32.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.910+0000 7f2f89ffb640 1 -- 172.21.15.71:0/1469233997 >> v1:172.21.15.137:6789/0 conn(0x7f2f8c1457f0 legacy=0x7f2f8c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:32.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.910+0000 7f2f89ffb640 1 -- 172.21.15.71:0/1469233997 >> v1:172.21.15.71:6789/0 conn(0x7f2f8c0a9d70 legacy=0x7f2f8c14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:09:32.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.911+0000 7f2f89ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:09:32.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.911+0000 7f2f89ffb640 1 monclient: found mon.noname-b 2024-03-29T11:09:32.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.911+0000 7f2f89ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:32.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.911+0000 7f2f89ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:09:32.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.911+0000 7f2f89ffb640 1 -- 172.21.15.71:0/1469233997 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f8c146340 con 0x7f2f8c13f650 2024-03-29T11:09:32.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.911+0000 7f2f89ffb640 10 monclient: _finish_auth 0 2024-03-29T11:09:32.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.911+0000 7f2f89ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:09:32.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.911+0000 7f2f89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:32.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.911+0000 7f2f89ffb640 1 -- 172.21.15.71:0/1469233997 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2f80001690 con 0x7f2f8c13f650 2024-03-29T11:09:32.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.911+0000 7f2f89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:32.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.911+0000 7f2f89ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:09:32.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.911+0000 7f2f89ffb640 10 monclient: dump: 2024-03-29T11:09:32.892 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:32.892 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:32.892 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:32.892 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:32.892 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:32.892 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:32.892 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:32.892 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:32.893 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:32.893 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:32.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.911+0000 7f2f91a16640 5 monclient: authenticate success, global_id 4613 2024-03-29T11:09:32.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.911+0000 7f2f91a16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:32.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.912+0000 7f2f89ffb640 1 -- 172.21.15.71:0/1469233997 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f80001860 con 0x7f2f8c13f650 2024-03-29T11:09:32.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.912+0000 7f2f89ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:32.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.912+0000 7f2f89ffb640 1 -- 172.21.15.71:0/1469233997 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2f80001cb0 con 0x7f2f8c13f650 2024-03-29T11:09:32.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.912+0000 7f2f89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:32.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.912+0000 7f2f89ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:32.896 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.912+0000 7f2f89ffb640 10 monclient: dump: 2024-03-29T11:09:32.896 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:32.896 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:32.896 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:32.896 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:32.896 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:32.896 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:32.896 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:32.896 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:32.896 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:32.896 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:32.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.912+0000 7f2f91a16640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:32.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.912+0000 7f2f91a16640 4 set_mon_vals no callback set 2024-03-29T11:09:32.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.913+0000 7f2f91a16640 10 monclient: shutdown 2024-03-29T11:09:32.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.913+0000 7f2f91a16640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:32.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.913+0000 7f2f91a16640 1 -- 172.21.15.71:0/1469233997 >> v1:172.21.15.71:6790/0 conn(0x7f2f8c13f650 legacy=0x7f2f8c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:32.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.913+0000 7f2f91a16640 1 -- 172.21.15.71:0/1469233997 shutdown_connections 2024-03-29T11:09:32.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.913+0000 7f2f91a16640 1 -- 172.21.15.71:0/1469233997 >> 172.21.15.71:0/1469233997 conn(0x7f2f8c066a00 msgr2=0x7f2f8c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:32.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.914+0000 7f2f91a16640 1 -- 172.21.15.71:0/1469233997 shutdown_connections 2024-03-29T11:09:32.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.914+0000 7f2f91a16640 1 -- 172.21.15.71:0/1469233997 wait complete. 2024-03-29T11:09:32.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.915+0000 7f2f91a16640 10 monclient: build_initial_monmap 2024-03-29T11:09:32.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.915+0000 7f2f91a16640 10 monclient: monmap: 2024-03-29T11:09:32.897 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:09:32.897 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:32.897 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:09:32.898 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:09:32.898 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:32.898 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:32.898 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:09:32.898 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:09:32.898 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:09:32.898 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:32.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.915+0000 7f2f91a16640 1 librados: starting msgr at 2024-03-29T11:09:32.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.915+0000 7f2f91a16640 1 librados: starting objecter 2024-03-29T11:09:32.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.915+0000 7f2f91a16640 1 Processor -- start 2024-03-29T11:09:32.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.915+0000 7f2f91a16640 1 -- start start 2024-03-29T11:09:32.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.915+0000 7f2f91a16640 1 librados: setting wanted keys 2024-03-29T11:09:32.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.915+0000 7f2f91a16640 1 librados: calling monclient init 2024-03-29T11:09:32.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.915+0000 7f2f91a16640 10 monclient: init 2024-03-29T11:09:32.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.916+0000 7f2f91a16640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:32.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.916+0000 7f2f91a16640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:09:32.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.916+0000 7f2f91a16640 10 monclient(hunting): picked mon.noname-b con 0x7f2f8c0a9d70 addr v1:172.21.15.137:6789/0 2024-03-29T11:09:32.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.916+0000 7f2f91a16640 10 monclient(hunting): picked mon.noname-a con 0x7f2f8c13f650 addr v1:172.21.15.71:6789/0 2024-03-29T11:09:32.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.916+0000 7f2f91a16640 10 monclient(hunting): picked mon.noname-c con 0x7f2f8c1457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:09:32.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.916+0000 7f2f91a16640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8c0703e0 con 0x7f2f8c13f650 2024-03-29T11:09:32.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.916+0000 7f2f91a16640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8c1dfc60 con 0x7f2f8c0a9d70 2024-03-29T11:09:32.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.916+0000 7f2f91a16640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8c071400 con 0x7f2f8c1457f0 2024-03-29T11:09:32.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.916+0000 7f2f91a16640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:32.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.916+0000 7f2f91a16640 10 monclient(hunting): authenticate will time out at 1329.174881s 2024-03-29T11:09:32.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.916+0000 7f2f8b7fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f2f8c1457f0 0x7f2f8c0b0650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:49224/0 (socket says 172.21.15.71:49224) 2024-03-29T11:09:32.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.916+0000 7f2f8b7fe640 1 -- 172.21.15.71:0/749475336 learned_addr learned my addr 172.21.15.71:0/749475336 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:09:32.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.917+0000 7f2f73fff640 10 client.?.objecter ms_handle_connect 0x7f2f8c1457f0 2024-03-29T11:09:32.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.917+0000 7f2f73fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:32.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.917+0000 7f2f73fff640 1 -- 172.21.15.71:0/749475336 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610865786 0 0) 0x7f2f8c071400 con 0x7f2f8c1457f0 2024-03-29T11:09:32.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.917+0000 7f2f73fff640 10 monclient(hunting): _negotiate 2024-03-29T11:09:32.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.917+0000 7f2f73fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:32.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.917+0000 7f2f73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:32.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.917+0000 7f2f73fff640 10 monclient(hunting): my global_id is 4616 2024-03-29T11:09:32.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.917+0000 7f2f73fff640 1 -- 172.21.15.71:0/749475336 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f78004eb0 con 0x7f2f8c1457f0 2024-03-29T11:09:32.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.917+0000 7f2f73fff640 1 -- 172.21.15.71:0/749475336 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1639419558 0 0) 0x7f2f78004eb0 con 0x7f2f8c1457f0 2024-03-29T11:09:32.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.917+0000 7f2f73fff640 1 -- 172.21.15.71:0/749475336 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f8c071400 con 0x7f2f8c1457f0 2024-03-29T11:09:32.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.917+0000 7f2f73fff640 1 -- 172.21.15.71:0/749475336 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2f84001c60 con 0x7f2f8c1457f0 2024-03-29T11:09:32.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.917+0000 7f2f73fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:32.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.917+0000 7f2f73fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:09:32.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.917+0000 7f2f73fff640 10 monclient(hunting): dump: 2024-03-29T11:09:32.905 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:32.905 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:32.905 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:32.906 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:32.906 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:32.906 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:32.906 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:32.906 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:32.906 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:32.906 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:32.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.917+0000 7f2f73fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:32.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.917+0000 7f2f73fff640 10 client.?.objecter ms_handle_connect 0x7f2f8c0a9d70 2024-03-29T11:09:32.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.917+0000 7f2f73fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:32.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.917+0000 7f2f73fff640 1 -- 172.21.15.71:0/749475336 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 43638402 0 0) 0x7f2f8c071400 con 0x7f2f8c1457f0 2024-03-29T11:09:32.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.917+0000 7f2f73fff640 1 -- 172.21.15.71:0/749475336 >> v1:172.21.15.137:6789/0 conn(0x7f2f8c0a9d70 legacy=0x7f2f8c071820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:32.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.917+0000 7f2f73fff640 1 -- 172.21.15.71:0/749475336 >> v1:172.21.15.71:6789/0 conn(0x7f2f8c13f650 legacy=0x7f2f8c06eda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:09:32.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.919+0000 7f2f73fff640 10 monclient: _finish_hunting 0 2024-03-29T11:09:32.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.919+0000 7f2f73fff640 1 monclient: found mon.c 2024-03-29T11:09:32.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.919+0000 7f2f73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:32.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.919+0000 7f2f73fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:32.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.919+0000 7f2f73fff640 1 -- 172.21.15.71:0/749475336 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f8c1df740 con 0x7f2f8c1457f0 2024-03-29T11:09:32.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.919+0000 7f2f73fff640 10 monclient: _finish_auth 0 2024-03-29T11:09:32.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.919+0000 7f2f73fff640 10 monclient: _check_auth_tickets 2024-03-29T11:09:32.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.919+0000 7f2f73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:32.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.919+0000 7f2f91a16640 5 monclient: authenticate success, global_id 4616 2024-03-29T11:09:32.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.919+0000 7f2f91a16640 10 monclient: _renew_subs 2024-03-29T11:09:32.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.919+0000 7f2f91a16640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:32.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.919+0000 7f2f91a16640 1 -- 172.21.15.71:0/749475336 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f8c06f4b0 con 0x7f2f8c1457f0 2024-03-29T11:09:32.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.919+0000 7f2f91a16640 10 client.4616.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:32.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.919+0000 7f2f91a16640 10 monclient: _renew_subs 2024-03-29T11:09:32.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.919+0000 7f2f91a16640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:32.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.919+0000 7f2f91a16640 1 -- 172.21.15.71:0/749475336 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f8c0b0d60 con 0x7f2f8c1457f0 2024-03-29T11:09:32.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.919+0000 7f2f91a16640 1 librados: init done 2024-03-29T11:09:32.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.919+0000 7f2f73fff640 1 -- 172.21.15.71:0/749475336 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f84003970 con 0x7f2f8c1457f0 2024-03-29T11:09:32.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.919+0000 7f2f73fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:32.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.919+0000 7f2f73fff640 1 -- 172.21.15.71:0/749475336 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2f84003d80 con 0x7f2f8c1457f0 2024-03-29T11:09:32.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.919+0000 7f2f73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:32.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.920+0000 7f2f73fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:32.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.920+0000 7f2f73fff640 10 monclient: dump: 2024-03-29T11:09:32.908 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:32.908 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:32.908 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:32.908 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:32.908 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:32.908 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:32.908 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:32.909 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:32.909 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:32.909 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:32.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.920+0000 7f2f897fa640 4 set_mon_vals no callback set 2024-03-29T11:09:32.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.920+0000 7f2f91a16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:32.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.920+0000 7f2f91a16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:32.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.920+0000 7f2f91a16640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:32.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.921+0000 7f2f91a16640 1 -- 172.21.15.71:0/749475336 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f6c005080 con 0x7f2f8c1457f0 2024-03-29T11:09:32.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.922+0000 7f2f73fff640 1 -- 172.21.15.71:0/749475336 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f2f84003ee0 con 0x7f2f8c1457f0 2024-03-29T11:09:32.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.922+0000 7f2f73fff640 1 -- 172.21.15.71:0/749475336 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f2f84084df0 con 0x7f2f8c1457f0 2024-03-29T11:09:32.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.922+0000 7f2f73fff640 10 client.4616.objecter ms_dispatch 0x7f2f8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:09:32.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.922+0000 7f2f73fff640 3 client.4616.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:09:32.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.922+0000 7f2f73fff640 3 client.4616.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:09:32.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.922+0000 7f2f73fff640 20 client.4616.objecter dump_active .. 0 homeless 2024-03-29T11:09:32.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.923+0000 7f2f8affd640 10 monclient: get_auth_request con 0x7f2f7806dc60 auth_method 0 2024-03-29T11:09:32.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.923+0000 7f2f73fff640 10 client.4616.objecter ms_handle_connect 0x7f2f7806dc60 2024-03-29T11:09:32.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.929+0000 7f2f73fff640 1 -- 172.21.15.71:0/749475336 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2f840506d0 con 0x7f2f8c1457f0 2024-03-29T11:09:32.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.929+0000 7f2f73fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:32.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:32.929+0000 7f2f73fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:33.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:33.236+0000 7f2f91a16640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:33.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:33.236+0000 7f2f91a16640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:33.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:33.236+0000 7f2f91a16640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:33.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:33.236+0000 7f2f91a16640 1 -- 172.21.15.71:0/749475336 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f6c005330 con 0x7f2f8c1457f0 2024-03-29T11:09:33.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:33.237+0000 7f2f73fff640 1 -- 172.21.15.71:0/749475336 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f2f84052600 con 0x7f2f8c1457f0 2024-03-29T11:09:33.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:33.237+0000 7f2f73fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:33.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:33.237+0000 7f2f73fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:33.215 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:09:33.215 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:09:33.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:33.240+0000 7f2f91a16640 10 librados: watch_flush enter 2024-03-29T11:09:33.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:33.240+0000 7f2f91a16640 10 librados: watch_flush exit 2024-03-29T11:09:33.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:33.240+0000 7f2f91a16640 20 client.4616.objecter shutdown clearing up homeless session... 2024-03-29T11:09:33.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:33.240+0000 7f2f91a16640 10 client.4616.objecter successfully canceled tick 2024-03-29T11:09:33.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:33.240+0000 7f2f91a16640 1 -- 172.21.15.71:0/749475336 >> v1:172.21.15.137:6816/973332665 conn(0x7f2f7806dc60 legacy=0x7f2f78070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:33.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:33.240+0000 7f2f91a16640 10 monclient: shutdown 2024-03-29T11:09:33.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:33.240+0000 7f2f91a16640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:33.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:33.240+0000 7f2f91a16640 1 -- 172.21.15.71:0/749475336 >> v1:172.21.15.71:6790/0 conn(0x7f2f8c1457f0 legacy=0x7f2f8c0b0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:33.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:33.240+0000 7f2f91a16640 1 -- 172.21.15.71:0/749475336 shutdown_connections 2024-03-29T11:09:33.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:33.240+0000 7f2f91a16640 1 -- 172.21.15.71:0/749475336 >> 172.21.15.71:0/749475336 conn(0x7f2f8c066a00 msgr2=0x7f2f8c13abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:33.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:33.240+0000 7f2f91a16640 1 -- 172.21.15.71:0/749475336 shutdown_connections 2024-03-29T11:09:33.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:33.240+0000 7f2f91a16640 1 -- 172.21.15.71:0/749475336 wait complete. 2024-03-29T11:09:33.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:33.240+0000 7f2f91a16640 1 librados: shutdown 2024-03-29T11:09:33.233 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:09:40.040 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:09:40.188 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.209+0000 7f26c66a8640 10 monclient: get_monmap_and_config 2024-03-29T11:09:40.188 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.209+0000 7f26c66a8640 10 monclient: build_initial_monmap 2024-03-29T11:09:40.188 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.209+0000 7f26c66a8640 10 monclient: monmap: 2024-03-29T11:09:40.188 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:09:40.188 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:40.188 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:09:40.210765+0000 2024-03-29T11:09:40.188 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:09:40.210765+0000 2024-03-29T11:09:40.188 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:40.188 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:40.189 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:09:40.189 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:09:40.189 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:09:40.189 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:40.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.211+0000 7f26c66a8640 1 Processor -- start 2024-03-29T11:09:40.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.211+0000 7f26c66a8640 1 -- start start 2024-03-29T11:09:40.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.211+0000 7f26c66a8640 10 monclient: init 2024-03-29T11:09:40.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.212+0000 7f26c66a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:40.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.212+0000 7f26c66a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:09:40.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.212+0000 7f26c66a8640 10 monclient(hunting): picked mon.noname-a con 0x7f26c00743e0 addr v1:172.21.15.71:6789/0 2024-03-29T11:09:40.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.212+0000 7f26c66a8640 10 monclient(hunting): picked mon.noname-b con 0x7f26c00747c0 addr v1:172.21.15.71:6790/0 2024-03-29T11:09:40.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.212+0000 7f26c66a8640 10 monclient(hunting): picked mon.noname-c con 0x7f26c01468a0 addr v1:172.21.15.137:6789/0 2024-03-29T11:09:40.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.212+0000 7f26c66a8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c00702d0 con 0x7f26c00743e0 2024-03-29T11:09:40.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.212+0000 7f26c66a8640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c0070540 con 0x7f26c01468a0 2024-03-29T11:09:40.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.212+0000 7f26c66a8640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c0070820 con 0x7f26c00747c0 2024-03-29T11:09:40.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.212+0000 7f26c66a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:40.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.212+0000 7f26c66a8640 10 monclient(hunting): authenticate will time out at 1336.470678s 2024-03-29T11:09:40.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.214+0000 7f26bf7fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f26c00747c0 0x7f26c0146190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:54162/0 (socket says 172.21.15.71:54162) 2024-03-29T11:09:40.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.214+0000 7f26bf7fe640 1 -- 172.21.15.71:0/2755488418 learned_addr learned my addr 172.21.15.71:0/2755488418 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:09:40.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.216+0000 7f26beffd640 1 -- 172.21.15.71:0/2755488418 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227501603 0 0) 0x7f26c0070820 con 0x7f26c00747c0 2024-03-29T11:09:40.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.216+0000 7f26beffd640 10 monclient(hunting): _negotiate 2024-03-29T11:09:40.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.216+0000 7f26beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:40.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.216+0000 7f26beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:40.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.216+0000 7f26beffd640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:09:40.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.216+0000 7f26beffd640 1 -- 172.21.15.71:0/2755488418 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26b0003520 con 0x7f26c00747c0 2024-03-29T11:09:40.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.216+0000 7f26beffd640 1 -- 172.21.15.71:0/2755488418 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870559008 0 0) 0x7f26c00702d0 con 0x7f26c00743e0 2024-03-29T11:09:40.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.217+0000 7f26beffd640 10 monclient(hunting): _negotiate 2024-03-29T11:09:40.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.217+0000 7f26beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:40.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.217+0000 7f26beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:40.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.217+0000 7f26beffd640 10 monclient(hunting): my global_id is 4566 2024-03-29T11:09:40.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.217+0000 7f26beffd640 1 -- 172.21.15.71:0/2755488418 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c0070820 con 0x7f26c00743e0 2024-03-29T11:09:40.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.217+0000 7f26beffd640 1 -- 172.21.15.71:0/2755488418 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 160546308 0 0) 0x7f26b0003520 con 0x7f26c00747c0 2024-03-29T11:09:40.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.217+0000 7f26beffd640 1 -- 172.21.15.71:0/2755488418 >> v1:172.21.15.137:6789/0 conn(0x7f26c01468a0 legacy=0x7f26c0151ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:40.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.217+0000 7f26beffd640 1 -- 172.21.15.71:0/2755488418 >> v1:172.21.15.71:6789/0 conn(0x7f26c00743e0 legacy=0x7f26c01434b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:40.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.218+0000 7f26beffd640 10 monclient: _finish_hunting 0 2024-03-29T11:09:40.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.218+0000 7f26beffd640 1 monclient: found mon.noname-b 2024-03-29T11:09:40.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.218+0000 7f26beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:40.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.218+0000 7f26beffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:09:40.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.218+0000 7f26beffd640 1 -- 172.21.15.71:0/2755488418 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c0149f90 con 0x7f26c00747c0 2024-03-29T11:09:40.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.218+0000 7f26beffd640 10 monclient: _finish_auth 0 2024-03-29T11:09:40.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.218+0000 7f26beffd640 10 monclient: _check_auth_tickets 2024-03-29T11:09:40.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.218+0000 7f26beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:40.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.218+0000 7f26c66a8640 5 monclient: authenticate success, global_id 4619 2024-03-29T11:09:40.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.218+0000 7f26c66a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:40.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.218+0000 7f26beffd640 1 -- 172.21.15.71:0/2755488418 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f26b4001690 con 0x7f26c00747c0 2024-03-29T11:09:40.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.218+0000 7f26beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:40.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.218+0000 7f26beffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:09:40.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.218+0000 7f26beffd640 10 monclient: dump: 2024-03-29T11:09:40.199 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:40.199 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:40.199 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:40.199 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:40.199 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:40.199 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:40.199 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:40.199 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:40.199 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:40.199 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:40.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.218+0000 7f26c66a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:40.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.218+0000 7f26beffd640 1 -- 172.21.15.71:0/2755488418 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26b40017f0 con 0x7f26c00747c0 2024-03-29T11:09:40.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.218+0000 7f26beffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:40.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.218+0000 7f26c66a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:40.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.218+0000 7f26c66a8640 4 set_mon_vals no callback set 2024-03-29T11:09:40.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.219+0000 7f26c66a8640 10 monclient: shutdown 2024-03-29T11:09:40.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.219+0000 7f26c66a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:40.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.219+0000 7f26c66a8640 1 -- 172.21.15.71:0/2755488418 >> v1:172.21.15.71:6790/0 conn(0x7f26c00747c0 legacy=0x7f26c0146190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:40.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.219+0000 7f26c66a8640 1 -- 172.21.15.71:0/2755488418 shutdown_connections 2024-03-29T11:09:40.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.219+0000 7f26c66a8640 1 -- 172.21.15.71:0/2755488418 >> 172.21.15.71:0/2755488418 conn(0x7f26c00b3ed0 msgr2=0x7f26c00b42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:40.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.220+0000 7f26c66a8640 1 -- 172.21.15.71:0/2755488418 shutdown_connections 2024-03-29T11:09:40.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.220+0000 7f26c66a8640 1 -- 172.21.15.71:0/2755488418 wait complete. 2024-03-29T11:09:40.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.221+0000 7f26c66a8640 10 monclient: build_initial_monmap 2024-03-29T11:09:40.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.221+0000 7f26c66a8640 10 monclient: monmap: 2024-03-29T11:09:40.201 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:09:40.201 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:40.201 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:09:40.201 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:09:40.201 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:40.201 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:40.201 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:09:40.202 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:09:40.202 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:09:40.202 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:40.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.221+0000 7f26c66a8640 1 librados: starting msgr at 2024-03-29T11:09:40.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.221+0000 7f26c66a8640 1 librados: starting objecter 2024-03-29T11:09:40.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.221+0000 7f26c66a8640 1 Processor -- start 2024-03-29T11:09:40.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.222+0000 7f26c66a8640 1 -- start start 2024-03-29T11:09:40.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.222+0000 7f26c66a8640 1 librados: setting wanted keys 2024-03-29T11:09:40.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.222+0000 7f26c66a8640 1 librados: calling monclient init 2024-03-29T11:09:40.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.222+0000 7f26c66a8640 10 monclient: init 2024-03-29T11:09:40.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.222+0000 7f26c66a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:40.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.222+0000 7f26c66a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:09:40.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.225+0000 7f26bffff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f26c00743e0 0x7f26c00adc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:33795/0 (socket says 172.21.15.71:33795) 2024-03-29T11:09:40.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.225+0000 7f26c66a8640 10 monclient(hunting): picked mon.noname-a con 0x7f26c00743e0 addr v1:172.21.15.71:6789/0 2024-03-29T11:09:40.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.225+0000 7f26c66a8640 10 monclient(hunting): picked mon.noname-b con 0x7f26c01468a0 addr v1:172.21.15.137:6789/0 2024-03-29T11:09:40.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.226+0000 7f26c66a8640 10 monclient(hunting): picked mon.noname-c con 0x7f26c00ad5e0 addr v1:172.21.15.71:6790/0 2024-03-29T11:09:40.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.226+0000 7f26c66a8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c014a630 con 0x7f26c00743e0 2024-03-29T11:09:40.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.226+0000 7f26c66a8640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c00ae3a0 con 0x7f26c01468a0 2024-03-29T11:09:40.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.226+0000 7f26c66a8640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c00ad150 con 0x7f26c00ad5e0 2024-03-29T11:09:40.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.226+0000 7f26c66a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:40.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.226+0000 7f26c66a8640 10 monclient(hunting): authenticate will time out at 1336.484177s 2024-03-29T11:09:40.206 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.226+0000 7f26bffff640 1 -- 172.21.15.71:0/711932004 learned_addr learned my addr 172.21.15.71:0/711932004 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:09:40.206 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.226+0000 7f26bcff9640 10 client.?.objecter ms_handle_connect 0x7f26c00743e0 2024-03-29T11:09:40.206 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.226+0000 7f26bcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:40.206 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.228+0000 7f26bcff9640 1 -- 172.21.15.71:0/711932004 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615940775 0 0) 0x7f26c014a630 con 0x7f26c00743e0 2024-03-29T11:09:40.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.228+0000 7f26bcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:09:40.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.228+0000 7f26bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:40.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.228+0000 7f26bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:40.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.228+0000 7f26bcff9640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:09:40.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.228+0000 7f26bcff9640 1 -- 172.21.15.71:0/711932004 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26a4004eb0 con 0x7f26c00743e0 2024-03-29T11:09:40.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.229+0000 7f26bcff9640 10 client.?.objecter ms_handle_connect 0x7f26c00ad5e0 2024-03-29T11:09:40.211 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.229+0000 7f26bcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:40.211 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.229+0000 7f26bcff9640 1 -- 172.21.15.71:0/711932004 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3121738752 0 0) 0x7f26a4004eb0 con 0x7f26c00743e0 2024-03-29T11:09:40.211 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.229+0000 7f26bcff9640 1 -- 172.21.15.71:0/711932004 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26c014a630 con 0x7f26c00743e0 2024-03-29T11:09:40.211 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.229+0000 7f26bcff9640 1 -- 172.21.15.71:0/711932004 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f26ac001a50 con 0x7f26c00743e0 2024-03-29T11:09:40.211 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.229+0000 7f26bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:40.211 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.229+0000 7f26bcff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:09:40.211 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.229+0000 7f26bcff9640 10 monclient(hunting): dump: 2024-03-29T11:09:40.211 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:40.211 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:40.211 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:40.211 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:40.212 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:40.212 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:40.212 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:40.212 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:40.212 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:40.212 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:40.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.229+0000 7f26bcff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:40.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.229+0000 7f26bcff9640 1 -- 172.21.15.71:0/711932004 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982246263 0 0) 0x7f26c00ad150 con 0x7f26c00ad5e0 2024-03-29T11:09:40.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.229+0000 7f26bcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:09:40.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.229+0000 7f26bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:40.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.229+0000 7f26bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:40.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.229+0000 7f26bcff9640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:09:40.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.229+0000 7f26bcff9640 1 -- 172.21.15.71:0/711932004 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26a4004eb0 con 0x7f26c00ad5e0 2024-03-29T11:09:40.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.229+0000 7f26bcff9640 10 client.?.objecter ms_handle_connect 0x7f26c01468a0 2024-03-29T11:09:40.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.229+0000 7f26bcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:40.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.229+0000 7f26bcff9640 1 -- 172.21.15.71:0/711932004 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3501633419 0 0) 0x7f26c00ae3a0 con 0x7f26c01468a0 2024-03-29T11:09:40.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.229+0000 7f26bcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:09:40.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.229+0000 7f26bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:40.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.229+0000 7f26bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:40.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.229+0000 7f26bcff9640 10 monclient(hunting): my global_id is 4504 2024-03-29T11:09:40.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.229+0000 7f26bcff9640 1 -- 172.21.15.71:0/711932004 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c00ad150 con 0x7f26c01468a0 2024-03-29T11:09:40.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.230+0000 7f26bcff9640 1 -- 172.21.15.71:0/711932004 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1328822920 0 0) 0x7f26a4004eb0 con 0x7f26c00ad5e0 2024-03-29T11:09:40.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.230+0000 7f26bcff9640 1 -- 172.21.15.71:0/711932004 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26c00ae3a0 con 0x7f26c00ad5e0 2024-03-29T11:09:40.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.230+0000 7f26bcff9640 1 -- 172.21.15.71:0/711932004 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f26b8001ac0 con 0x7f26c00ad5e0 2024-03-29T11:09:40.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.230+0000 7f26bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:40.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.230+0000 7f26bcff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:40.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.230+0000 7f26bcff9640 10 monclient(hunting): dump: 2024-03-29T11:09:40.213 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:40.214 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:40.214 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:40.214 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:40.214 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:40.214 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:40.214 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:40.214 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:40.214 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:40.214 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:40.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.230+0000 7f26bcff9640 1 -- 172.21.15.71:0/711932004 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3676063367 0 0) 0x7f26c00ad150 con 0x7f26c01468a0 2024-03-29T11:09:40.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.230+0000 7f26bcff9640 1 -- 172.21.15.71:0/711932004 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26a4004eb0 con 0x7f26c01468a0 2024-03-29T11:09:40.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.230+0000 7f26bcff9640 1 -- 172.21.15.71:0/711932004 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f26b40014d0 con 0x7f26c01468a0 2024-03-29T11:09:40.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.230+0000 7f26bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:40.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.230+0000 7f26bcff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:40.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.230+0000 7f26bcff9640 10 monclient(hunting): dump: 2024-03-29T11:09:40.215 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:40.215 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:40.215 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:40.215 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:40.215 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:40.215 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:40.215 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:40.215 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:40.215 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:40.215 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:40.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.230+0000 7f26bcff9640 1 -- 172.21.15.71:0/711932004 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3963225275 0 0) 0x7f26c00ae3a0 con 0x7f26c00ad5e0 2024-03-29T11:09:40.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.230+0000 7f26bcff9640 1 -- 172.21.15.71:0/711932004 >> v1:172.21.15.137:6789/0 conn(0x7f26c01468a0 legacy=0x7f26c01db6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:40.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.230+0000 7f26bcff9640 1 -- 172.21.15.71:0/711932004 >> v1:172.21.15.71:6789/0 conn(0x7f26c00743e0 legacy=0x7f26c00adc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:40.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.230+0000 7f26bcff9640 10 monclient: _finish_hunting 0 2024-03-29T11:09:40.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.230+0000 7f26bcff9640 1 monclient: found mon.c 2024-03-29T11:09:40.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.230+0000 7f26bcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:40.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.230+0000 7f26bcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:40.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.230+0000 7f26bcff9640 1 -- 172.21.15.71:0/711932004 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c0124ee0 con 0x7f26c00ad5e0 2024-03-29T11:09:40.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.231+0000 7f26bcff9640 10 monclient: _finish_auth 0 2024-03-29T11:09:40.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.231+0000 7f26bcff9640 10 monclient: _check_auth_tickets 2024-03-29T11:09:40.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.231+0000 7f26bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:40.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.231+0000 7f26c66a8640 5 monclient: authenticate success, global_id 4622 2024-03-29T11:09:40.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.231+0000 7f26c66a8640 10 monclient: _renew_subs 2024-03-29T11:09:40.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.231+0000 7f26c66a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:40.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.231+0000 7f26c66a8640 1 -- 172.21.15.71:0/711932004 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c0125220 con 0x7f26c00ad5e0 2024-03-29T11:09:40.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.231+0000 7f26c66a8640 10 client.4622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:40.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.231+0000 7f26c66a8640 10 monclient: _renew_subs 2024-03-29T11:09:40.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.231+0000 7f26c66a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:40.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.231+0000 7f26c66a8640 1 -- 172.21.15.71:0/711932004 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26c00ace90 con 0x7f26c00ad5e0 2024-03-29T11:09:40.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.231+0000 7f26c66a8640 1 librados: init done 2024-03-29T11:09:40.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.231+0000 7f26bcff9640 1 -- 172.21.15.71:0/711932004 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26b8001c20 con 0x7f26c00ad5e0 2024-03-29T11:09:40.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.231+0000 7f26bcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:40.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.231+0000 7f26be7fc640 4 set_mon_vals no callback set 2024-03-29T11:09:40.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.232+0000 7f26bcff9640 1 -- 172.21.15.71:0/711932004 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f26b80039a0 con 0x7f26c00ad5e0 2024-03-29T11:09:40.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.232+0000 7f26bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:40.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.232+0000 7f26bcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:40.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.232+0000 7f26bcff9640 10 monclient: dump: 2024-03-29T11:09:40.217 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:40.217 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:40.217 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:40.217 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:40.217 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:40.217 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:40.217 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:40.218 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:40.218 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:40.218 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:40.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.233+0000 7f26bcff9640 1 -- 172.21.15.71:0/711932004 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f26b8003b00 con 0x7f26c00ad5e0 2024-03-29T11:09:40.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.233+0000 7f26bcff9640 1 -- 172.21.15.71:0/711932004 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f26b80838c0 con 0x7f26c00ad5e0 2024-03-29T11:09:40.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.233+0000 7f26bcff9640 10 client.4622.objecter ms_dispatch 0x7f26c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:09:40.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.234+0000 7f26bcff9640 3 client.4622.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:09:40.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.234+0000 7f26bcff9640 3 client.4622.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:09:40.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.234+0000 7f26bcff9640 20 client.4622.objecter dump_active .. 0 homeless 2024-03-29T11:09:40.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.234+0000 7f26c66a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:40.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.234+0000 7f26c66a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:40.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.234+0000 7f26c66a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:40.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.234+0000 7f26c66a8640 1 -- 172.21.15.71:0/711932004 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26a0005080 con 0x7f26c00ad5e0 2024-03-29T11:09:40.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.235+0000 7f26bffff640 10 monclient: get_auth_request con 0x7f26a406e110 auth_method 0 2024-03-29T11:09:40.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.237+0000 7f26bcff9640 10 client.4622.objecter ms_handle_connect 0x7f26a406e110 2024-03-29T11:09:40.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.243+0000 7f26bcff9640 1 -- 172.21.15.71:0/711932004 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f26b804f030 con 0x7f26c00ad5e0 2024-03-29T11:09:40.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.243+0000 7f26bcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:40.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.243+0000 7f26bcff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:40.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.570+0000 7f26c66a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:40.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.570+0000 7f26c66a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:40.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.570+0000 7f26c66a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:40.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.570+0000 7f26c66a8640 1 -- 172.21.15.71:0/711932004 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26a0005330 con 0x7f26c00ad5e0 2024-03-29T11:09:40.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.573+0000 7f26bcff9640 1 -- 172.21.15.71:0/711932004 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f26b8082f70 con 0x7f26c00ad5e0 2024-03-29T11:09:40.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.573+0000 7f26bcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:40.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.573+0000 7f26bcff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:40.553 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:09:40.553 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:09:40.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.578+0000 7f26c66a8640 10 librados: watch_flush enter 2024-03-29T11:09:40.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.578+0000 7f26c66a8640 10 librados: watch_flush exit 2024-03-29T11:09:40.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.578+0000 7f26c66a8640 20 client.4622.objecter shutdown clearing up homeless session... 2024-03-29T11:09:40.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.578+0000 7f26c66a8640 10 client.4622.objecter successfully canceled tick 2024-03-29T11:09:40.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.578+0000 7f26c66a8640 1 -- 172.21.15.71:0/711932004 >> v1:172.21.15.137:6816/973332665 conn(0x7f26a406e110 legacy=0x7f26a40705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:40.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.578+0000 7f26c66a8640 10 monclient: shutdown 2024-03-29T11:09:40.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.578+0000 7f26c66a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:40.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.578+0000 7f26c66a8640 1 -- 172.21.15.71:0/711932004 >> v1:172.21.15.71:6790/0 conn(0x7f26c00ad5e0 legacy=0x7f26c01e61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:40.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.578+0000 7f26c66a8640 1 -- 172.21.15.71:0/711932004 shutdown_connections 2024-03-29T11:09:40.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.579+0000 7f26c66a8640 1 -- 172.21.15.71:0/711932004 >> 172.21.15.71:0/711932004 conn(0x7f26c00b3ed0 msgr2=0x7f26c00b3770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:40.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.579+0000 7f26c66a8640 1 -- 172.21.15.71:0/711932004 shutdown_connections 2024-03-29T11:09:40.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.579+0000 7f26c66a8640 1 -- 172.21.15.71:0/711932004 wait complete. 2024-03-29T11:09:40.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:40.579+0000 7f26c66a8640 1 librados: shutdown 2024-03-29T11:09:40.577 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:09:47.384 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:09:47.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.553+0000 7f75c02ff640 10 monclient: get_monmap_and_config 2024-03-29T11:09:47.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.553+0000 7f75c02ff640 10 monclient: build_initial_monmap 2024-03-29T11:09:47.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.553+0000 7f75c02ff640 10 monclient: monmap: 2024-03-29T11:09:47.532 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:09:47.532 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:47.532 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:09:47.553951+0000 2024-03-29T11:09:47.532 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:09:47.553951+0000 2024-03-29T11:09:47.532 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:47.532 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:47.532 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:09:47.533 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:09:47.533 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:09:47.533 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:47.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.553+0000 7f75c02ff640 1 Processor -- start 2024-03-29T11:09:47.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.554+0000 7f75c02ff640 1 -- start start 2024-03-29T11:09:47.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.554+0000 7f75c02ff640 10 monclient: init 2024-03-29T11:09:47.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.555+0000 7f75c02ff640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:47.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.555+0000 7f75c02ff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:09:47.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.557+0000 7f75be09b640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f75b81457f0 0x7f75b813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:60449/0 (socket says 172.21.15.71:60449) 2024-03-29T11:09:47.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.557+0000 7f75c02ff640 10 monclient(hunting): picked mon.noname-a con 0x7f75b81457f0 addr v1:172.21.15.71:6789/0 2024-03-29T11:09:47.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.557+0000 7f75c02ff640 10 monclient(hunting): picked mon.noname-b con 0x7f75b813f650 addr v1:172.21.15.71:6790/0 2024-03-29T11:09:47.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.557+0000 7f75c02ff640 10 monclient(hunting): picked mon.noname-c con 0x7f75b80a9d70 addr v1:172.21.15.137:6789/0 2024-03-29T11:09:47.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.557+0000 7f75c02ff640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b80aaef0 con 0x7f75b81457f0 2024-03-29T11:09:47.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.557+0000 7f75c02ff640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b8145bd0 con 0x7f75b80a9d70 2024-03-29T11:09:47.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.557+0000 7f75c02ff640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b8145eb0 con 0x7f75b813f650 2024-03-29T11:09:47.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.557+0000 7f75c02ff640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:47.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.557+0000 7f75c02ff640 10 monclient(hunting): authenticate will time out at 1343.815927s 2024-03-29T11:09:47.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.557+0000 7f75be09b640 1 -- 172.21.15.71:0/1475080550 learned_addr learned my addr 172.21.15.71:0/1475080550 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:09:47.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.558+0000 7f75bd099640 1 -- 172.21.15.71:0/1475080550 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2358319845 0 0) 0x7f75b8145eb0 con 0x7f75b813f650 2024-03-29T11:09:47.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.558+0000 7f75bd099640 10 monclient(hunting): _negotiate 2024-03-29T11:09:47.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.558+0000 7f75bd099640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:47.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.558+0000 7f75bd099640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:47.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.558+0000 7f75bd099640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:09:47.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.558+0000 7f75bd099640 1 -- 172.21.15.71:0/1475080550 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75a4003520 con 0x7f75b813f650 2024-03-29T11:09:47.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.558+0000 7f75bd099640 1 -- 172.21.15.71:0/1475080550 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2049960462 0 0) 0x7f75b80aaef0 con 0x7f75b81457f0 2024-03-29T11:09:47.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.558+0000 7f75bd099640 10 monclient(hunting): _negotiate 2024-03-29T11:09:47.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.558+0000 7f75bd099640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:47.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.558+0000 7f75bd099640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:47.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.558+0000 7f75bd099640 10 monclient(hunting): my global_id is 4572 2024-03-29T11:09:47.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.558+0000 7f75bd099640 1 -- 172.21.15.71:0/1475080550 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75b8145eb0 con 0x7f75b81457f0 2024-03-29T11:09:47.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.558+0000 7f75bd099640 1 -- 172.21.15.71:0/1475080550 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1748992427 0 0) 0x7f75b8145eb0 con 0x7f75b81457f0 2024-03-29T11:09:47.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.558+0000 7f75bd099640 1 -- 172.21.15.71:0/1475080550 >> v1:172.21.15.71:6790/0 conn(0x7f75b813f650 legacy=0x7f75b80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:47.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.558+0000 7f75bd099640 1 -- 172.21.15.71:0/1475080550 >> v1:172.21.15.137:6789/0 conn(0x7f75b80a9d70 legacy=0x7f75b814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:09:47.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.562+0000 7f75bd099640 10 monclient: _finish_hunting 0 2024-03-29T11:09:47.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.562+0000 7f75bd099640 1 monclient: found mon.noname-a 2024-03-29T11:09:47.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.562+0000 7f75bd099640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:47.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.562+0000 7f75bd099640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:09:47.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.562+0000 7f75bd099640 1 -- 172.21.15.71:0/1475080550 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b8146340 con 0x7f75b81457f0 2024-03-29T11:09:47.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.562+0000 7f75bd099640 10 monclient: _finish_auth 0 2024-03-29T11:09:47.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.562+0000 7f75bd099640 10 monclient: _check_auth_tickets 2024-03-29T11:09:47.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.562+0000 7f75bd099640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:47.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.562+0000 7f75bd099640 1 -- 172.21.15.71:0/1475080550 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f75b40015a0 con 0x7f75b81457f0 2024-03-29T11:09:47.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.562+0000 7f75bd099640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:47.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.562+0000 7f75bd099640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:09:47.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.562+0000 7f75bd099640 10 monclient: dump: 2024-03-29T11:09:47.545 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:47.545 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:47.546 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:47.546 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:47.546 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:47.546 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:47.546 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:47.546 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:47.546 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:47.546 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:47.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.562+0000 7f75c02ff640 5 monclient: authenticate success, global_id 4572 2024-03-29T11:09:47.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.562+0000 7f75c02ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:47.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.562+0000 7f75bd099640 1 -- 172.21.15.71:0/1475080550 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75b4001700 con 0x7f75b81457f0 2024-03-29T11:09:47.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.562+0000 7f75bd099640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:47.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.562+0000 7f75bd099640 1 -- 172.21.15.71:0/1475080550 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f75b4001b10 con 0x7f75b81457f0 2024-03-29T11:09:47.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.562+0000 7f75c02ff640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:47.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.562+0000 7f75c02ff640 4 set_mon_vals no callback set 2024-03-29T11:09:47.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.562+0000 7f75bd099640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:47.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.562+0000 7f75bd099640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:09:47.547 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.562+0000 7f75bd099640 10 monclient: dump: 2024-03-29T11:09:47.547 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:47.547 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:47.548 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:47.548 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:47.548 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:47.548 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:47.548 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:47.548 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:47.548 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:47.548 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:47.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.563+0000 7f75c02ff640 10 monclient: shutdown 2024-03-29T11:09:47.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.563+0000 7f75c02ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:47.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.563+0000 7f75c02ff640 1 -- 172.21.15.71:0/1475080550 >> v1:172.21.15.71:6789/0 conn(0x7f75b81457f0 legacy=0x7f75b813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:47.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.564+0000 7f75c02ff640 1 -- 172.21.15.71:0/1475080550 shutdown_connections 2024-03-29T11:09:47.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.564+0000 7f75c02ff640 1 -- 172.21.15.71:0/1475080550 >> 172.21.15.71:0/1475080550 conn(0x7f75b8066a00 msgr2=0x7f75b8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:47.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.566+0000 7f75c02ff640 1 -- 172.21.15.71:0/1475080550 shutdown_connections 2024-03-29T11:09:47.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.566+0000 7f75c02ff640 1 -- 172.21.15.71:0/1475080550 wait complete. 2024-03-29T11:09:47.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.567+0000 7f75c02ff640 10 monclient: build_initial_monmap 2024-03-29T11:09:47.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.567+0000 7f75c02ff640 10 monclient: monmap: 2024-03-29T11:09:47.561 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:09:47.561 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:47.562 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:09:47.562 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:09:47.562 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:47.562 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:47.562 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:09:47.562 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:09:47.562 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:09:47.562 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:47.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.567+0000 7f75c02ff640 1 librados: starting msgr at 2024-03-29T11:09:47.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.568+0000 7f75c02ff640 1 librados: starting objecter 2024-03-29T11:09:47.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.568+0000 7f75c02ff640 1 Processor -- start 2024-03-29T11:09:47.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.568+0000 7f75c02ff640 1 -- start start 2024-03-29T11:09:47.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.568+0000 7f75c02ff640 1 librados: setting wanted keys 2024-03-29T11:09:47.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.568+0000 7f75c02ff640 1 librados: calling monclient init 2024-03-29T11:09:47.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.568+0000 7f75c02ff640 10 monclient: init 2024-03-29T11:09:47.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.568+0000 7f75c02ff640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:47.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.568+0000 7f75c02ff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:09:47.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.568+0000 7f75c02ff640 10 monclient(hunting): picked mon.noname-b con 0x7f75b80a9d70 addr v1:172.21.15.137:6789/0 2024-03-29T11:09:47.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.568+0000 7f75c02ff640 10 monclient(hunting): picked mon.noname-c con 0x7f75b813f650 addr v1:172.21.15.71:6790/0 2024-03-29T11:09:47.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.569+0000 7f75bd89a640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f75b813f650 0x7f75b8130cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:55048/0 (socket says 172.21.15.71:55048) 2024-03-29T11:09:47.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.571+0000 7f75be89c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f75b81457f0 0x7f75b81e29f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:32985/0 (socket says 172.21.15.71:32985) 2024-03-29T11:09:47.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.571+0000 7f75bd89a640 1 -- 172.21.15.71:0/3629942288 learned_addr learned my addr 172.21.15.71:0/3629942288 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:09:47.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.572+0000 7f75c02ff640 10 monclient(hunting): picked mon.noname-a con 0x7f75b81457f0 addr v1:172.21.15.71:6789/0 2024-03-29T11:09:47.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.572+0000 7f75c02ff640 1 -- 172.21.15.71:0/3629942288 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b81360c0 con 0x7f75b81457f0 2024-03-29T11:09:47.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.572+0000 7f75c02ff640 1 -- 172.21.15.71:0/3629942288 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b8134540 con 0x7f75b80a9d70 2024-03-29T11:09:47.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.572+0000 7f75c02ff640 1 -- 172.21.15.71:0/3629942288 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b81d7800 con 0x7f75b813f650 2024-03-29T11:09:47.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.572+0000 7f75c02ff640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:47.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.572+0000 7f75c02ff640 10 monclient(hunting): authenticate will time out at 1343.830413s 2024-03-29T11:09:47.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.572+0000 7f75aeffd640 10 client.?.objecter ms_handle_connect 0x7f75b813f650 2024-03-29T11:09:47.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.572+0000 7f75aeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:47.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.572+0000 7f75aeffd640 10 client.?.objecter ms_handle_connect 0x7f75b81457f0 2024-03-29T11:09:47.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.572+0000 7f75aeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:47.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.572+0000 7f75aeffd640 1 -- 172.21.15.71:0/3629942288 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411417419 0 0) 0x7f75b81360c0 con 0x7f75b81457f0 2024-03-29T11:09:47.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.572+0000 7f75aeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:09:47.565 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.572+0000 7f75aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:47.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.572+0000 7f75aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:47.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.572+0000 7f75aeffd640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:09:47.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.572+0000 7f75aeffd640 1 -- 172.21.15.71:0/3629942288 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f759c004eb0 con 0x7f75b81457f0 2024-03-29T11:09:47.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.572+0000 7f75aeffd640 1 -- 172.21.15.71:0/3629942288 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2475770813 0 0) 0x7f75b81d7800 con 0x7f75b813f650 2024-03-29T11:09:47.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.572+0000 7f75aeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:09:47.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.572+0000 7f75aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:47.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.572+0000 7f75aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:47.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.572+0000 7f75aeffd640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:09:47.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.572+0000 7f75aeffd640 1 -- 172.21.15.71:0/3629942288 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75b81360c0 con 0x7f75b813f650 2024-03-29T11:09:47.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.574+0000 7f75aeffd640 1 -- 172.21.15.71:0/3629942288 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3267160376 0 0) 0x7f75b81360c0 con 0x7f75b813f650 2024-03-29T11:09:47.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.574+0000 7f75aeffd640 1 -- 172.21.15.71:0/3629942288 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75b81d7800 con 0x7f75b813f650 2024-03-29T11:09:47.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.574+0000 7f75aeffd640 10 client.?.objecter ms_handle_connect 0x7f75b80a9d70 2024-03-29T11:09:47.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.574+0000 7f75aeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:47.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.574+0000 7f75aeffd640 1 -- 172.21.15.71:0/3629942288 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f75a8001a70 con 0x7f75b813f650 2024-03-29T11:09:47.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.574+0000 7f75aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:47.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.574+0000 7f75aeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:09:47.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.574+0000 7f75aeffd640 10 monclient(hunting): dump: 2024-03-29T11:09:47.568 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:47.568 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:47.568 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:47.568 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:47.568 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:47.568 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:47.568 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:47.568 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:47.568 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:47.568 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:47.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.574+0000 7f75aeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:47.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.574+0000 7f75aeffd640 1 -- 172.21.15.71:0/3629942288 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3521814615 0 0) 0x7f75b81d7800 con 0x7f75b813f650 2024-03-29T11:09:47.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.574+0000 7f75aeffd640 1 -- 172.21.15.71:0/3629942288 >> v1:172.21.15.137:6789/0 conn(0x7f75b80a9d70 legacy=0x7f75b81d70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:47.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.574+0000 7f75aeffd640 1 -- 172.21.15.71:0/3629942288 >> v1:172.21.15.71:6789/0 conn(0x7f75b81457f0 legacy=0x7f75b81e29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:47.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.574+0000 7f75aeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:09:47.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.574+0000 7f75aeffd640 1 monclient: found mon.c 2024-03-29T11:09:47.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.574+0000 7f75aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:47.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.574+0000 7f75aeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:47.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.574+0000 7f75aeffd640 1 -- 172.21.15.71:0/3629942288 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b81364e0 con 0x7f75b813f650 2024-03-29T11:09:47.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.575+0000 7f75aeffd640 10 monclient: _finish_auth 0 2024-03-29T11:09:47.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.575+0000 7f75aeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:09:47.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.575+0000 7f75aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:47.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.575+0000 7f75c02ff640 5 monclient: authenticate success, global_id 4628 2024-03-29T11:09:47.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.575+0000 7f75c02ff640 10 monclient: _renew_subs 2024-03-29T11:09:47.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.575+0000 7f75c02ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:47.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.575+0000 7f75c02ff640 1 -- 172.21.15.71:0/3629942288 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75b8136730 con 0x7f75b813f650 2024-03-29T11:09:47.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.575+0000 7f75c02ff640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:47.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.575+0000 7f75c02ff640 10 monclient: _renew_subs 2024-03-29T11:09:47.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.575+0000 7f75c02ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:47.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.575+0000 7f75c02ff640 1 -- 172.21.15.71:0/3629942288 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75b8134ee0 con 0x7f75b813f650 2024-03-29T11:09:47.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.575+0000 7f75c02ff640 1 librados: init done 2024-03-29T11:09:47.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.577+0000 7f75acff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:47.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.577+0000 7f75aeffd640 1 -- 172.21.15.71:0/3629942288 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75a8001bd0 con 0x7f75b813f650 2024-03-29T11:09:47.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.577+0000 7f75acff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:47.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.577+0000 7f75acff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:47.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.577+0000 7f75acff9640 1 -- 172.21.15.71:0/3629942288 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75a0005080 con 0x7f75b813f650 2024-03-29T11:09:47.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.578+0000 7f75aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:47.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.578+0000 7f75aeffd640 1 -- 172.21.15.71:0/3629942288 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f75a8001d80 con 0x7f75b813f650 2024-03-29T11:09:47.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.578+0000 7f75aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:47.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.578+0000 7f75aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:47.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.578+0000 7f75aeffd640 10 monclient: dump: 2024-03-29T11:09:47.571 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:47.571 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:47.571 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:47.571 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:47.571 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:47.571 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:47.571 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:47.572 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:47.572 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:47.572 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:47.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.578+0000 7f75aeffd640 1 -- 172.21.15.71:0/3629942288 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f75a8003b00 con 0x7f75b813f650 2024-03-29T11:09:47.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.579+0000 7f75be09b640 10 monclient: get_auth_request con 0x7f759c06de60 auth_method 0 2024-03-29T11:09:47.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.579+0000 7f75aeffd640 1 -- 172.21.15.71:0/3629942288 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f75a8084a30 con 0x7f75b813f650 2024-03-29T11:09:47.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.579+0000 7f75aeffd640 10 client.4628.objecter ms_dispatch 0x7f75b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:09:47.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.579+0000 7f75aeffd640 3 client.4628.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:09:47.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.579+0000 7f75aeffd640 3 client.4628.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:09:47.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.579+0000 7f75aeffd640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-29T11:09:47.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.579+0000 7f75aeffd640 10 client.4628.objecter ms_handle_connect 0x7f759c06de60 2024-03-29T11:09:47.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.579+0000 7f75affff640 4 set_mon_vals no callback set 2024-03-29T11:09:47.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.587+0000 7f75aeffd640 1 -- 172.21.15.71:0/3629942288 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f75a8050420 con 0x7f75b813f650 2024-03-29T11:09:47.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.587+0000 7f75aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:47.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.587+0000 7f75aeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:47.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.918+0000 7f75acff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:47.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.918+0000 7f75acff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:47.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.918+0000 7f75acff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:47.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.918+0000 7f75acff9640 1 -- 172.21.15.71:0/3629942288 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75a0005330 con 0x7f75b813f650 2024-03-29T11:09:47.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.920+0000 7f75aeffd640 1 -- 172.21.15.71:0/3629942288 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f75a8052350 con 0x7f75b813f650 2024-03-29T11:09:47.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.920+0000 7f75aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:47.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.920+0000 7f75aeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:47.899 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:09:47.899 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:09:47.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.924+0000 7f75acff9640 10 librados: watch_flush enter 2024-03-29T11:09:47.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.924+0000 7f75acff9640 10 librados: watch_flush exit 2024-03-29T11:09:47.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.924+0000 7f75acff9640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-29T11:09:47.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.924+0000 7f75acff9640 10 client.4628.objecter successfully canceled tick 2024-03-29T11:09:47.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.926+0000 7f75acff9640 1 -- 172.21.15.71:0/3629942288 >> v1:172.21.15.137:6816/973332665 conn(0x7f759c06de60 legacy=0x7f759c070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:47.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.926+0000 7f75acff9640 10 monclient: shutdown 2024-03-29T11:09:47.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.926+0000 7f75acff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:47.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.926+0000 7f75acff9640 1 -- 172.21.15.71:0/3629942288 >> v1:172.21.15.71:6790/0 conn(0x7f75b813f650 legacy=0x7f75b8130cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:47.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.926+0000 7f75acff9640 1 -- 172.21.15.71:0/3629942288 shutdown_connections 2024-03-29T11:09:47.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.926+0000 7f75acff9640 1 -- 172.21.15.71:0/3629942288 >> 172.21.15.71:0/3629942288 conn(0x7f75b8066a00 msgr2=0x7f75b80b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:47.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.926+0000 7f75acff9640 1 -- 172.21.15.71:0/3629942288 shutdown_connections 2024-03-29T11:09:47.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.927+0000 7f75acff9640 1 -- 172.21.15.71:0/3629942288 wait complete. 2024-03-29T11:09:47.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:47.927+0000 7f75acff9640 1 librados: shutdown 2024-03-29T11:09:47.922 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:09:54.730 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:09:54.882 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.904+0000 7f18b21dc640 10 monclient: get_monmap_and_config 2024-03-29T11:09:54.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.904+0000 7f18b21dc640 10 monclient: build_initial_monmap 2024-03-29T11:09:54.883 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.904+0000 7f18b21dc640 10 monclient: monmap: 2024-03-29T11:09:54.883 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:09:54.883 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:54.883 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:09:54.905275+0000 2024-03-29T11:09:54.883 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:09:54.905275+0000 2024-03-29T11:09:54.883 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:54.883 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:54.883 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:09:54.883 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:09:54.883 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:09:54.883 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:54.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.906+0000 7f18b21dc640 1 Processor -- start 2024-03-29T11:09:54.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.907+0000 7f18b21dc640 1 -- start start 2024-03-29T11:09:54.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.907+0000 7f18b21dc640 10 monclient: init 2024-03-29T11:09:54.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.908+0000 7f18b21dc640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:54.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.908+0000 7f18b21dc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:09:54.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.908+0000 7f18b21dc640 10 monclient(hunting): picked mon.noname-c con 0x7f18ac0b4820 addr v1:172.21.15.137:6789/0 2024-03-29T11:09:54.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.908+0000 7f18b21dc640 10 monclient(hunting): picked mon.noname-b con 0x7f18ac0a9ce0 addr v1:172.21.15.71:6790/0 2024-03-29T11:09:54.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.908+0000 7f18b21dc640 10 monclient(hunting): picked mon.noname-a con 0x7f18ac0b3cd0 addr v1:172.21.15.71:6789/0 2024-03-29T11:09:54.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.908+0000 7f18b21dc640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ac0b4180 con 0x7f18ac0b3cd0 2024-03-29T11:09:54.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.908+0000 7f18b21dc640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ac0aaf60 con 0x7f18ac0b4820 2024-03-29T11:09:54.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.908+0000 7f18b21dc640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ac13c200 con 0x7f18ac0a9ce0 2024-03-29T11:09:54.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.908+0000 7f18b21dc640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:54.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.908+0000 7f18b21dc640 10 monclient(hunting): authenticate will time out at 1351.166816s 2024-03-29T11:09:54.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.909+0000 7f18b09d9640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f18ac0a9ce0 0x7f18ac145770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:58476/0 (socket says 172.21.15.71:58476) 2024-03-29T11:09:54.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.909+0000 7f18b09d9640 1 -- 172.21.15.71:0/1896322228 learned_addr learned my addr 172.21.15.71:0/1896322228 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:09:54.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.909+0000 7f18a3fff640 1 -- 172.21.15.71:0/1896322228 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197098657 0 0) 0x7f18ac13c200 con 0x7f18ac0a9ce0 2024-03-29T11:09:54.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.909+0000 7f18a3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:09:54.889 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.909+0000 7f18a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:54.889 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.909+0000 7f18a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:54.889 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.909+0000 7f18a3fff640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:09:54.889 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.909+0000 7f18a3fff640 1 -- 172.21.15.71:0/1896322228 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1898003520 con 0x7f18ac0a9ce0 2024-03-29T11:09:54.889 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.909+0000 7f18a3fff640 1 -- 172.21.15.71:0/1896322228 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4000812342 0 0) 0x7f1898003520 con 0x7f18ac0a9ce0 2024-03-29T11:09:54.889 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.909+0000 7f18a3fff640 1 -- 172.21.15.71:0/1896322228 >> v1:172.21.15.137:6789/0 conn(0x7f18ac0b4820 legacy=0x7f18ac13baf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:09:54.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.911+0000 7f18a3fff640 1 -- 172.21.15.71:0/1896322228 >> v1:172.21.15.71:6789/0 conn(0x7f18ac0b3cd0 legacy=0x7f18ac14d7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:09:54.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.911+0000 7f18a3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:09:54.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.911+0000 7f18a3fff640 1 monclient: found mon.noname-b 2024-03-29T11:09:54.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.911+0000 7f18a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:54.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.911+0000 7f18a3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:09:54.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.911+0000 7f18a3fff640 1 -- 172.21.15.71:0/1896322228 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ac145e80 con 0x7f18ac0a9ce0 2024-03-29T11:09:54.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.911+0000 7f18a3fff640 10 monclient: _finish_auth 0 2024-03-29T11:09:54.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.911+0000 7f18a3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:09:54.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.911+0000 7f18a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:54.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.911+0000 7f18a3fff640 1 -- 172.21.15.71:0/1896322228 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f18a8001670 con 0x7f18ac0a9ce0 2024-03-29T11:09:54.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.911+0000 7f18a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:54.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.911+0000 7f18a3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:09:54.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.911+0000 7f18a3fff640 10 monclient: dump: 2024-03-29T11:09:54.893 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:54.893 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:54.893 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:54.893 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:54.893 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:54.893 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:54.893 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:54.893 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:54.893 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:54.893 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:54.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.912+0000 7f18a3fff640 1 -- 172.21.15.71:0/1896322228 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18a80018d0 con 0x7f18ac0a9ce0 2024-03-29T11:09:54.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.912+0000 7f18a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:54.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.912+0000 7f18a3fff640 1 -- 172.21.15.71:0/1896322228 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f18a8001d20 con 0x7f18ac0a9ce0 2024-03-29T11:09:54.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.912+0000 7f18a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:54.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.912+0000 7f18a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:54.894 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.912+0000 7f18a3fff640 10 monclient: dump: 2024-03-29T11:09:54.894 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:54.894 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:54.894 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:54.894 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:54.894 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:54.894 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:54.894 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:54.894 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:54.894 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:54.894 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:54.894 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.912+0000 7f18b21dc640 5 monclient: authenticate success, global_id 4631 2024-03-29T11:09:54.894 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.912+0000 7f18b21dc640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:54.894 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.912+0000 7f18b21dc640 4 set_mon_vals no callback set 2024-03-29T11:09:54.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.913+0000 7f18b21dc640 10 monclient: shutdown 2024-03-29T11:09:54.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.913+0000 7f18b21dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:54.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.913+0000 7f18b21dc640 1 -- 172.21.15.71:0/1896322228 >> v1:172.21.15.71:6790/0 conn(0x7f18ac0a9ce0 legacy=0x7f18ac145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:54.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.913+0000 7f18b21dc640 1 -- 172.21.15.71:0/1896322228 shutdown_connections 2024-03-29T11:09:54.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.913+0000 7f18b21dc640 1 -- 172.21.15.71:0/1896322228 >> 172.21.15.71:0/1896322228 conn(0x7f18ac06d070 msgr2=0x7f18ac066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:54.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.914+0000 7f18b21dc640 1 -- 172.21.15.71:0/1896322228 shutdown_connections 2024-03-29T11:09:54.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.914+0000 7f18b21dc640 1 -- 172.21.15.71:0/1896322228 wait complete. 2024-03-29T11:09:54.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.915+0000 7f18b21dc640 10 monclient: build_initial_monmap 2024-03-29T11:09:54.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.915+0000 7f18b21dc640 10 monclient: monmap: 2024-03-29T11:09:54.897 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:09:54.897 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:54.897 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:09:54.897 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:09:54.897 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:54.897 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:54.897 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:09:54.897 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:09:54.898 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:09:54.898 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:54.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.916+0000 7f18b21dc640 1 librados: starting msgr at 2024-03-29T11:09:54.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.916+0000 7f18b21dc640 1 librados: starting objecter 2024-03-29T11:09:54.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.916+0000 7f18b21dc640 1 Processor -- start 2024-03-29T11:09:54.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.916+0000 7f18b21dc640 1 -- start start 2024-03-29T11:09:54.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.916+0000 7f18b21dc640 1 librados: setting wanted keys 2024-03-29T11:09:54.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.916+0000 7f18b21dc640 1 librados: calling monclient init 2024-03-29T11:09:54.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.916+0000 7f18b21dc640 10 monclient: init 2024-03-29T11:09:54.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.917+0000 7f18b21dc640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:54.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.917+0000 7f18b21dc640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:09:54.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.917+0000 7f18b11da640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f18ac0a9ce0 0x7f18ac13df00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:32978/0 (socket says 172.21.15.71:32978) 2024-03-29T11:09:54.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.917+0000 7f18b11da640 1 -- 172.21.15.71:0/702607944 learned_addr learned my addr 172.21.15.71:0/702607944 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:09:54.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.917+0000 7f18b21dc640 10 monclient(hunting): picked mon.noname-c con 0x7f18ac0a9ce0 addr v1:172.21.15.71:6790/0 2024-03-29T11:09:54.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.918+0000 7f18b21dc640 10 monclient(hunting): picked mon.noname-a con 0x7f18ac0b3cd0 addr v1:172.21.15.71:6789/0 2024-03-29T11:09:54.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.918+0000 7f18b21dc640 10 monclient(hunting): picked mon.noname-b con 0x7f18ac0b4820 addr v1:172.21.15.137:6789/0 2024-03-29T11:09:54.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.918+0000 7f18b21dc640 1 -- 172.21.15.71:0/702607944 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ac13f460 con 0x7f18ac0b3cd0 2024-03-29T11:09:54.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.918+0000 7f18b21dc640 1 -- 172.21.15.71:0/702607944 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ac13d3a0 con 0x7f18ac0b4820 2024-03-29T11:09:54.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.918+0000 7f18b21dc640 1 -- 172.21.15.71:0/702607944 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ac13e610 con 0x7f18ac0a9ce0 2024-03-29T11:09:54.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.918+0000 7f18b21dc640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:54.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.918+0000 7f18b21dc640 10 monclient(hunting): authenticate will time out at 1351.176696s 2024-03-29T11:09:54.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.919+0000 7f18a1ffb640 10 client.?.objecter ms_handle_connect 0x7f18ac0a9ce0 2024-03-29T11:09:54.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.919+0000 7f18a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:54.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.919+0000 7f18a1ffb640 1 -- 172.21.15.71:0/702607944 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685205172 0 0) 0x7f18ac13e610 con 0x7f18ac0a9ce0 2024-03-29T11:09:54.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.919+0000 7f18a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:09:54.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.919+0000 7f18a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:54.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.919+0000 7f18a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:54.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.919+0000 7f18a1ffb640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:09:54.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.920+0000 7f18a1ffb640 1 -- 172.21.15.71:0/702607944 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1890004eb0 con 0x7f18ac0a9ce0 2024-03-29T11:09:54.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.920+0000 7f18a1ffb640 1 -- 172.21.15.71:0/702607944 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2906154243 0 0) 0x7f1890004eb0 con 0x7f18ac0a9ce0 2024-03-29T11:09:54.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.920+0000 7f18a1ffb640 1 -- 172.21.15.71:0/702607944 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18ac13e610 con 0x7f18ac0a9ce0 2024-03-29T11:09:54.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.920+0000 7f18a1ffb640 1 -- 172.21.15.71:0/702607944 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f18a4001b50 con 0x7f18ac0a9ce0 2024-03-29T11:09:54.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.920+0000 7f18a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:54.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.920+0000 7f18a1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:09:54.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.920+0000 7f18a1ffb640 10 monclient(hunting): dump: 2024-03-29T11:09:54.901 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:54.901 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:54.901 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:54.901 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:54.901 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:54.901 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:54.901 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:54.901 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:54.901 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:54.901 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:54.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.920+0000 7f18a1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:54.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.920+0000 7f18a1ffb640 1 -- 172.21.15.71:0/702607944 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1528275851 0 0) 0x7f18ac13e610 con 0x7f18ac0a9ce0 2024-03-29T11:09:54.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.920+0000 7f18a1ffb640 1 -- 172.21.15.71:0/702607944 >> v1:172.21.15.137:6789/0 conn(0x7f18ac0b4820 legacy=0x7f18ac131ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:09:54.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.922+0000 7f18a1ffb640 1 -- 172.21.15.71:0/702607944 >> v1:172.21.15.71:6789/0 conn(0x7f18ac0b3cd0 legacy=0x7f18ac1d7180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:09:54.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.923+0000 7f18a1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:09:54.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.923+0000 7f18a1ffb640 1 monclient: found mon.c 2024-03-29T11:09:54.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.923+0000 7f18a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:54.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.923+0000 7f18a1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:54.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.923+0000 7f18a1ffb640 1 -- 172.21.15.71:0/702607944 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ac13cfe0 con 0x7f18ac0a9ce0 2024-03-29T11:09:54.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.923+0000 7f18a1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:09:54.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.923+0000 7f18a1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:09:54.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.923+0000 7f18a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:54.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.923+0000 7f18b21dc640 5 monclient: authenticate success, global_id 4634 2024-03-29T11:09:54.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.923+0000 7f18b21dc640 10 monclient: _renew_subs 2024-03-29T11:09:54.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.923+0000 7f18b21dc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:54.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.923+0000 7f18b21dc640 1 -- 172.21.15.71:0/702607944 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18ac1d7890 con 0x7f18ac0a9ce0 2024-03-29T11:09:54.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.923+0000 7f18b21dc640 10 client.4634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:54.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.924+0000 7f18b21dc640 10 monclient: _renew_subs 2024-03-29T11:09:54.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.924+0000 7f18b21dc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:54.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.924+0000 7f18b21dc640 1 -- 172.21.15.71:0/702607944 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18ac13da10 con 0x7f18ac0a9ce0 2024-03-29T11:09:54.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.924+0000 7f18b21dc640 1 librados: init done 2024-03-29T11:09:54.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.925+0000 7f18a1ffb640 1 -- 172.21.15.71:0/702607944 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18a4001cb0 con 0x7f18ac0a9ce0 2024-03-29T11:09:54.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.925+0000 7f18a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:54.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.925+0000 7f18a1ffb640 1 -- 172.21.15.71:0/702607944 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f18a4004dc0 con 0x7f18ac0a9ce0 2024-03-29T11:09:54.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.925+0000 7f18a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:54.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.925+0000 7f18a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:54.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.925+0000 7f18a1ffb640 10 monclient: dump: 2024-03-29T11:09:54.907 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:09:54.907 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:09:54.907 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:54.907 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:09:54.907 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:54.907 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:09:54.907 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:09:54.907 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:09:54.907 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:09:54.907 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:09:54.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.925+0000 7f18b21dc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:54.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.925+0000 7f18b21dc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:54.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.925+0000 7f18b21dc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:54.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.925+0000 7f18b21dc640 1 -- 172.21.15.71:0/702607944 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18ac04f730 con 0x7f18ac0a9ce0 2024-03-29T11:09:54.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.927+0000 7f18a37fe640 4 set_mon_vals no callback set 2024-03-29T11:09:54.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.928+0000 7f18a1ffb640 1 -- 172.21.15.71:0/702607944 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f18a401ae50 con 0x7f18ac0a9ce0 2024-03-29T11:09:54.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.929+0000 7f18a1ffb640 1 -- 172.21.15.71:0/702607944 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f18a4084e20 con 0x7f18ac0a9ce0 2024-03-29T11:09:54.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.929+0000 7f18a1ffb640 10 client.4634.objecter ms_dispatch 0x7f18ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:09:54.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.929+0000 7f18a1ffb640 3 client.4634.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:09:54.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.929+0000 7f18a1ffb640 3 client.4634.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:09:54.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.929+0000 7f18b09d9640 10 monclient: get_auth_request con 0x7f189006dc90 auth_method 0 2024-03-29T11:09:54.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.929+0000 7f18a1ffb640 20 client.4634.objecter dump_active .. 0 homeless 2024-03-29T11:09:54.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.929+0000 7f18a1ffb640 10 client.4634.objecter ms_handle_connect 0x7f189006dc90 2024-03-29T11:09:54.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.935+0000 7f18a1ffb640 1 -- 172.21.15.71:0/702607944 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f18a4050530 con 0x7f18ac0a9ce0 2024-03-29T11:09:54.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.936+0000 7f18a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:54.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:54.936+0000 7f18a1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:55.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:55.254+0000 7f18b21dc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:55.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:55.254+0000 7f18b21dc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:55.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:55.254+0000 7f18b21dc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:09:55.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:55.254+0000 7f18b21dc640 1 -- 172.21.15.71:0/702607944 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18ac13d170 con 0x7f18ac0a9ce0 2024-03-29T11:09:55.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:55.255+0000 7f18a1ffb640 1 -- 172.21.15.71:0/702607944 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f18ac13d170 con 0x7f18ac0a9ce0 2024-03-29T11:09:55.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:55.255+0000 7f18a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:55.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:55.255+0000 7f18a1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:55.234 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:09:55.234 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:09:55.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:55.259+0000 7f18b21dc640 10 librados: watch_flush enter 2024-03-29T11:09:55.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:55.259+0000 7f18b21dc640 10 librados: watch_flush exit 2024-03-29T11:09:55.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:55.259+0000 7f18b21dc640 20 client.4634.objecter shutdown clearing up homeless session... 2024-03-29T11:09:55.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:55.259+0000 7f18b21dc640 10 client.4634.objecter successfully canceled tick 2024-03-29T11:09:55.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:55.259+0000 7f18b21dc640 1 -- 172.21.15.71:0/702607944 >> v1:172.21.15.137:6816/973332665 conn(0x7f189006dc90 legacy=0x7f1890070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:55.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:55.260+0000 7f18b21dc640 10 monclient: shutdown 2024-03-29T11:09:55.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:55.260+0000 7f18b21dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:55.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:55.260+0000 7f18b21dc640 1 -- 172.21.15.71:0/702607944 >> v1:172.21.15.71:6790/0 conn(0x7f18ac0a9ce0 legacy=0x7f18ac13df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:55.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:55.260+0000 7f18b21dc640 1 -- 172.21.15.71:0/702607944 shutdown_connections 2024-03-29T11:09:55.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:55.260+0000 7f18b21dc640 1 -- 172.21.15.71:0/702607944 >> 172.21.15.71:0/702607944 conn(0x7f18ac06d070 msgr2=0x7f18ac0b5630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:55.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:55.260+0000 7f18b21dc640 1 -- 172.21.15.71:0/702607944 shutdown_connections 2024-03-29T11:09:55.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:55.260+0000 7f18b21dc640 1 -- 172.21.15.71:0/702607944 wait complete. 2024-03-29T11:09:55.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:09:55.260+0000 7f18b21dc640 1 librados: shutdown 2024-03-29T11:09:55.255 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:10:02.062 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:10:02.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.240+0000 7fd104c5f640 10 monclient: get_monmap_and_config 2024-03-29T11:10:02.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.240+0000 7fd104c5f640 10 monclient: build_initial_monmap 2024-03-29T11:10:02.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.240+0000 7fd104c5f640 10 monclient: monmap: 2024-03-29T11:10:02.221 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:10:02.221 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:02.221 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:10:02.242797+0000 2024-03-29T11:10:02.221 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:10:02.242797+0000 2024-03-29T11:10:02.221 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:02.221 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:02.221 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:10:02.222 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:10:02.222 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:10:02.222 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:02.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.241+0000 7fd104c5f640 1 Processor -- start 2024-03-29T11:10:02.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.242+0000 7fd104c5f640 1 -- start start 2024-03-29T11:10:02.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.242+0000 7fd104c5f640 10 monclient: init 2024-03-29T11:10:02.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.243+0000 7fd104c5f640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:02.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.243+0000 7fd104c5f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:10:02.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.243+0000 7fd104c5f640 10 monclient(hunting): picked mon.noname-a con 0x7fd1000b0500 addr v1:172.21.15.71:6789/0 2024-03-29T11:10:02.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.243+0000 7fd104c5f640 10 monclient(hunting): picked mon.noname-b con 0x7fd1000b08e0 addr v1:172.21.15.71:6790/0 2024-03-29T11:10:02.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.243+0000 7fd104c5f640 10 monclient(hunting): picked mon.noname-c con 0x7fd1000aeca0 addr v1:172.21.15.137:6789/0 2024-03-29T11:10:02.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.243+0000 7fd104c5f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd100140620 con 0x7fd1000b0500 2024-03-29T11:10:02.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.243+0000 7fd104c5f640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd10013daf0 con 0x7fd1000aeca0 2024-03-29T11:10:02.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.243+0000 7fd104c5f640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1000ae180 con 0x7fd1000b08e0 2024-03-29T11:10:02.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.243+0000 7fd0fe59c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fd1000b0500 0x7fd10013ff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:33940/0 (socket says 172.21.15.71:33940) 2024-03-29T11:10:02.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.243+0000 7fd0fe59c640 1 -- 172.21.15.71:0/914673999 learned_addr learned my addr 172.21.15.71:0/914673999 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:10:02.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.243+0000 7fd104c5f640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:02.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.243+0000 7fd104c5f640 10 monclient(hunting): authenticate will time out at 1358.502652s 2024-03-29T11:10:02.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.243+0000 7fd0fd59a640 1 -- 172.21.15.71:0/914673999 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1530331924 0 0) 0x7fd1000ae180 con 0x7fd1000b08e0 2024-03-29T11:10:02.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.243+0000 7fd0fd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:10:02.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.243+0000 7fd0fd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:02.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.243+0000 7fd0fd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:02.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.243+0000 7fd0fd59a640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:10:02.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.243+0000 7fd0fd59a640 1 -- 172.21.15.71:0/914673999 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0e8003520 con 0x7fd1000b08e0 2024-03-29T11:10:02.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.244+0000 7fd0fd59a640 1 -- 172.21.15.71:0/914673999 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1679720342 0 0) 0x7fd0e8003520 con 0x7fd1000b08e0 2024-03-29T11:10:02.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.244+0000 7fd0fd59a640 1 -- 172.21.15.71:0/914673999 >> v1:172.21.15.137:6789/0 conn(0x7fd1000aeca0 legacy=0x7fd10014b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:10:02.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.244+0000 7fd0fd59a640 1 -- 172.21.15.71:0/914673999 >> v1:172.21.15.71:6789/0 conn(0x7fd1000b0500 legacy=0x7fd10013ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:02.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.244+0000 7fd0fd59a640 10 monclient: _finish_hunting 0 2024-03-29T11:10:02.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.244+0000 7fd0fd59a640 1 monclient: found mon.noname-b 2024-03-29T11:10:02.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.244+0000 7fd0fd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:02.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.244+0000 7fd0fd59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:10:02.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.244+0000 7fd0fd59a640 1 -- 172.21.15.71:0/914673999 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1000accf0 con 0x7fd1000b08e0 2024-03-29T11:10:02.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.244+0000 7fd0fd59a640 10 monclient: _finish_auth 0 2024-03-29T11:10:02.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.244+0000 7fd0fd59a640 10 monclient: _check_auth_tickets 2024-03-29T11:10:02.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.244+0000 7fd0fd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:02.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.244+0000 7fd104c5f640 5 monclient: authenticate success, global_id 4637 2024-03-29T11:10:02.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.244+0000 7fd104c5f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:02.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.244+0000 7fd0fd59a640 1 -- 172.21.15.71:0/914673999 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fd0f0001690 con 0x7fd1000b08e0 2024-03-29T11:10:02.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.244+0000 7fd0fd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:02.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.244+0000 7fd0fd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:10:02.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.244+0000 7fd0fd59a640 10 monclient: dump: 2024-03-29T11:10:02.238 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:02.238 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:02.238 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:02.238 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:02.238 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:02.238 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:02.239 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:02.239 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:02.239 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:02.239 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:02.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.244+0000 7fd0fd59a640 1 -- 172.21.15.71:0/914673999 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd0f0001960 con 0x7fd1000b08e0 2024-03-29T11:10:02.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.244+0000 7fd0fd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:02.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.244+0000 7fd0fd59a640 1 -- 172.21.15.71:0/914673999 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fd0f0001cf0 con 0x7fd1000b08e0 2024-03-29T11:10:02.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.244+0000 7fd0fd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:02.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.244+0000 7fd0fd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:10:02.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.244+0000 7fd0fd59a640 10 monclient: dump: 2024-03-29T11:10:02.239 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:02.240 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:02.240 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:02.240 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:02.240 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:02.240 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:02.241 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:02.241 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:02.241 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:02.241 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:02.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.244+0000 7fd104c5f640 10 monclient: get_monmap_and_config success 2024-03-29T11:10:02.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.244+0000 7fd104c5f640 4 set_mon_vals no callback set 2024-03-29T11:10:02.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.245+0000 7fd104c5f640 10 monclient: shutdown 2024-03-29T11:10:02.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.245+0000 7fd104c5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:02.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.245+0000 7fd104c5f640 1 -- 172.21.15.71:0/914673999 >> v1:172.21.15.71:6790/0 conn(0x7fd1000b08e0 legacy=0x7fd10013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:02.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.245+0000 7fd104c5f640 1 -- 172.21.15.71:0/914673999 shutdown_connections 2024-03-29T11:10:02.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.245+0000 7fd104c5f640 1 -- 172.21.15.71:0/914673999 >> 172.21.15.71:0/914673999 conn(0x7fd10006b660 msgr2=0x7fd10006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:02.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.245+0000 7fd104c5f640 1 -- 172.21.15.71:0/914673999 shutdown_connections 2024-03-29T11:10:02.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.245+0000 7fd104c5f640 1 -- 172.21.15.71:0/914673999 wait complete. 2024-03-29T11:10:02.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.247+0000 7fd104c5f640 10 monclient: build_initial_monmap 2024-03-29T11:10:02.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.247+0000 7fd104c5f640 10 monclient: monmap: 2024-03-29T11:10:02.243 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:10:02.243 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:10:02.243 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:10:02.243 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:10:02.243 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:02.243 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:02.243 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:10:02.243 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:10:02.243 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:10:02.243 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:02.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.247+0000 7fd104c5f640 1 librados: starting msgr at 2024-03-29T11:10:02.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.247+0000 7fd104c5f640 1 librados: starting objecter 2024-03-29T11:10:02.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.247+0000 7fd104c5f640 1 Processor -- start 2024-03-29T11:10:02.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.248+0000 7fd104c5f640 1 -- start start 2024-03-29T11:10:02.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.248+0000 7fd104c5f640 1 librados: setting wanted keys 2024-03-29T11:10:02.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.248+0000 7fd104c5f640 1 librados: calling monclient init 2024-03-29T11:10:02.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.248+0000 7fd104c5f640 10 monclient: init 2024-03-29T11:10:02.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.248+0000 7fd104c5f640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:02.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.249+0000 7fd104c5f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:10:02.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.249+0000 7fd104c5f640 10 monclient(hunting): picked mon.noname-c con 0x7fd1000aeca0 addr v1:172.21.15.71:6790/0 2024-03-29T11:10:02.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.249+0000 7fd104c5f640 10 monclient(hunting): picked mon.noname-a con 0x7fd1000b0500 addr v1:172.21.15.71:6789/0 2024-03-29T11:10:02.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.249+0000 7fd0fe59c640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fd1000aeca0 0x7fd10006e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:32896/0 (socket says 172.21.15.71:32896) 2024-03-29T11:10:02.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.249+0000 7fd0fe59c640 1 -- 172.21.15.71:0/347379704 learned_addr learned my addr 172.21.15.71:0/347379704 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:10:02.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.249+0000 7fd104c5f640 10 monclient(hunting): picked mon.noname-b con 0x7fd100070c40 addr v1:172.21.15.137:6789/0 2024-03-29T11:10:02.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.249+0000 7fd104c5f640 1 -- 172.21.15.71:0/347379704 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd10006e470 con 0x7fd1000b0500 2024-03-29T11:10:02.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.249+0000 7fd0eeffd640 10 client.?.objecter ms_handle_connect 0x7fd1000aeca0 2024-03-29T11:10:02.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.249+0000 7fd0eeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:02.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.249+0000 7fd104c5f640 1 -- 172.21.15.71:0/347379704 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd10006df50 con 0x7fd100070c40 2024-03-29T11:10:02.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.249+0000 7fd104c5f640 1 -- 172.21.15.71:0/347379704 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1000aca60 con 0x7fd1000aeca0 2024-03-29T11:10:02.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.249+0000 7fd104c5f640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:02.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.249+0000 7fd104c5f640 10 monclient(hunting): authenticate will time out at 1358.508752s 2024-03-29T11:10:02.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.249+0000 7fd0eeffd640 10 client.?.objecter ms_handle_connect 0x7fd1000b0500 2024-03-29T11:10:02.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.249+0000 7fd0eeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:02.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.249+0000 7fd0eeffd640 10 client.?.objecter ms_handle_connect 0x7fd100070c40 2024-03-29T11:10:02.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.249+0000 7fd0eeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:02.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.250+0000 7fd0eeffd640 1 -- 172.21.15.71:0/347379704 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3216866282 0 0) 0x7fd10006e470 con 0x7fd1000b0500 2024-03-29T11:10:02.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.250+0000 7fd0eeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:10:02.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.250+0000 7fd0eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:02.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.250+0000 7fd0eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:02.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.250+0000 7fd0eeffd640 10 monclient(hunting): my global_id is 4581 2024-03-29T11:10:02.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.250+0000 7fd0eeffd640 1 -- 172.21.15.71:0/347379704 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0e0004eb0 con 0x7fd1000b0500 2024-03-29T11:10:02.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.250+0000 7fd0eeffd640 1 -- 172.21.15.71:0/347379704 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1566498174 0 0) 0x7fd10006df50 con 0x7fd100070c40 2024-03-29T11:10:02.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.250+0000 7fd0eeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:10:02.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.250+0000 7fd0eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:02.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.250+0000 7fd0eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:02.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.250+0000 7fd0eeffd640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:10:02.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.250+0000 7fd0eeffd640 1 -- 172.21.15.71:0/347379704 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd10006e470 con 0x7fd100070c40 2024-03-29T11:10:02.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.250+0000 7fd0eeffd640 1 -- 172.21.15.71:0/347379704 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607631825 0 0) 0x7fd1000aca60 con 0x7fd1000aeca0 2024-03-29T11:10:02.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.250+0000 7fd0eeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:10:02.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.250+0000 7fd0eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:02.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.250+0000 7fd0eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:02.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.250+0000 7fd0eeffd640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:10:02.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.250+0000 7fd0eeffd640 1 -- 172.21.15.71:0/347379704 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd10006df50 con 0x7fd1000aeca0 2024-03-29T11:10:02.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 1 -- 172.21.15.71:0/347379704 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1964566644 0 0) 0x7fd10006df50 con 0x7fd1000aeca0 2024-03-29T11:10:02.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 1 -- 172.21.15.71:0/347379704 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1000aca60 con 0x7fd1000aeca0 2024-03-29T11:10:02.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 1 -- 172.21.15.71:0/347379704 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1866045215 0 0) 0x7fd10006e470 con 0x7fd100070c40 2024-03-29T11:10:02.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 1 -- 172.21.15.71:0/347379704 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd10006df50 con 0x7fd100070c40 2024-03-29T11:10:02.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 1 -- 172.21.15.71:0/347379704 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fd0f8001b30 con 0x7fd100070c40 2024-03-29T11:10:02.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:02.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:10:02.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 10 monclient(hunting): dump: 2024-03-29T11:10:02.248 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:02.248 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:02.248 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:02.248 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:02.248 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:02.249 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:02.249 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:02.249 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:02.249 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:02.249 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:02.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:10:02.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 1 -- 172.21.15.71:0/347379704 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fd0f40018b0 con 0x7fd1000aeca0 2024-03-29T11:10:02.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:02.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:10:02.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 10 monclient(hunting): dump: 2024-03-29T11:10:02.250 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:02.250 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:02.250 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:02.250 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:02.250 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:02.250 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:02.250 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:02.250 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:02.251 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:02.251 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:02.251 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 1 -- 172.21.15.71:0/347379704 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2856174197 0 0) 0x7fd0e0004eb0 con 0x7fd1000b0500 2024-03-29T11:10:02.251 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 1 -- 172.21.15.71:0/347379704 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd10006e470 con 0x7fd1000b0500 2024-03-29T11:10:02.251 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 1 -- 172.21.15.71:0/347379704 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fd0f0001650 con 0x7fd1000b0500 2024-03-29T11:10:02.251 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:02.251 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:10:02.251 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 10 monclient(hunting): dump: 2024-03-29T11:10:02.251 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:02.252 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:02.252 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:02.252 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:02.252 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:02.252 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:02.252 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:02.252 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:02.252 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:02.252 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:02.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 1 -- 172.21.15.71:0/347379704 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 761335091 0 0) 0x7fd10006df50 con 0x7fd100070c40 2024-03-29T11:10:02.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 1 -- 172.21.15.71:0/347379704 >> v1:172.21.15.71:6790/0 conn(0x7fd1000aeca0 legacy=0x7fd10006e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:02.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 1 -- 172.21.15.71:0/347379704 >> v1:172.21.15.71:6789/0 conn(0x7fd1000b0500 legacy=0x7fd10013b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:02.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:10:02.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 1 monclient: found mon.b 2024-03-29T11:10:02.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:02.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:10:02.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 1 -- 172.21.15.71:0/347379704 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd10013bf80 con 0x7fd100070c40 2024-03-29T11:10:02.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 10 monclient: _finish_auth 0 2024-03-29T11:10:02.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:10:02.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd0eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:02.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd104c5f640 5 monclient: authenticate success, global_id 4507 2024-03-29T11:10:02.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd104c5f640 10 monclient: _renew_subs 2024-03-29T11:10:02.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd104c5f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:10:02.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd104c5f640 1 -- 172.21.15.71:0/347379704 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd10013c1f0 con 0x7fd100070c40 2024-03-29T11:10:02.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd104c5f640 10 client.4507.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:10:02.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd104c5f640 10 monclient: _renew_subs 2024-03-29T11:10:02.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd104c5f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:10:02.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd104c5f640 1 -- 172.21.15.71:0/347379704 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd10013d5e0 con 0x7fd100070c40 2024-03-29T11:10:02.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.251+0000 7fd104c5f640 1 librados: init done 2024-03-29T11:10:02.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.253+0000 7fd0eeffd640 1 -- 172.21.15.71:0/347379704 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd0f8002040 con 0x7fd100070c40 2024-03-29T11:10:02.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.253+0000 7fd0eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:02.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.253+0000 7fd0eeffd640 1 -- 172.21.15.71:0/347379704 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fd0f8003b10 con 0x7fd100070c40 2024-03-29T11:10:02.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.253+0000 7fd0eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:02.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.253+0000 7fd0eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:10:02.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.253+0000 7fd0eeffd640 10 monclient: dump: 2024-03-29T11:10:02.257 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:02.257 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:02.257 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:02.257 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:02.257 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:02.257 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:02.257 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:02.257 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:02.258 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:02.258 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:02.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.253+0000 7fd0fcd99640 4 set_mon_vals no callback set 2024-03-29T11:10:02.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.253+0000 7fd104c5f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:02.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.254+0000 7fd104c5f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:02.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.254+0000 7fd104c5f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:10:02.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.254+0000 7fd104c5f640 1 -- 172.21.15.71:0/347379704 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e4005080 con 0x7fd100070c40 2024-03-29T11:10:02.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.254+0000 7fd0eeffd640 1 -- 172.21.15.71:0/347379704 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7fd0f8019ae0 con 0x7fd100070c40 2024-03-29T11:10:02.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.256+0000 7fd0eeffd640 1 -- 172.21.15.71:0/347379704 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7fd0f8083a30 con 0x7fd100070c40 2024-03-29T11:10:02.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.256+0000 7fd0eeffd640 10 client.4507.objecter ms_dispatch 0x7fd100000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:10:02.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.256+0000 7fd0eeffd640 3 client.4507.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:10:02.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.256+0000 7fd0eeffd640 3 client.4507.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:10:02.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.256+0000 7fd0eeffd640 20 client.4507.objecter dump_active .. 0 homeless 2024-03-29T11:10:02.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.257+0000 7fd0fe59c640 10 monclient: get_auth_request con 0x7fd0e006e0e0 auth_method 0 2024-03-29T11:10:02.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.257+0000 7fd0eeffd640 10 client.4507.objecter ms_handle_connect 0x7fd0e006e0e0 2024-03-29T11:10:02.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.266+0000 7fd0eeffd640 1 -- 172.21.15.71:0/347379704 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd0f804f310 con 0x7fd100070c40 2024-03-29T11:10:02.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.266+0000 7fd0eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:02.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.266+0000 7fd0eeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:10:02.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.632+0000 7fd104c5f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:02.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.632+0000 7fd104c5f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:02.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.632+0000 7fd104c5f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:10:02.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.632+0000 7fd104c5f640 1 -- 172.21.15.71:0/347379704 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0e4005330 con 0x7fd100070c40 2024-03-29T11:10:02.619 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.635+0000 7fd0eeffd640 1 -- 172.21.15.71:0/347379704 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7fd0f8051240 con 0x7fd100070c40 2024-03-29T11:10:02.619 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.635+0000 7fd0eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:02.619 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.635+0000 7fd0eeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:10:02.619 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:10:02.619 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:10:02.621 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.638+0000 7fd104c5f640 10 librados: watch_flush enter 2024-03-29T11:10:02.621 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.638+0000 7fd104c5f640 10 librados: watch_flush exit 2024-03-29T11:10:02.621 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.638+0000 7fd104c5f640 20 client.4507.objecter shutdown clearing up homeless session... 2024-03-29T11:10:02.621 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.638+0000 7fd104c5f640 10 client.4507.objecter successfully canceled tick 2024-03-29T11:10:02.621 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.639+0000 7fd104c5f640 1 -- 172.21.15.71:0/347379704 >> v1:172.21.15.137:6816/973332665 conn(0x7fd0e006e0e0 legacy=0x7fd0e00705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:02.621 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.639+0000 7fd104c5f640 10 monclient: shutdown 2024-03-29T11:10:02.621 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.639+0000 7fd104c5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:02.621 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.639+0000 7fd104c5f640 1 -- 172.21.15.71:0/347379704 >> v1:172.21.15.137:6789/0 conn(0x7fd100070c40 legacy=0x7fd1001e2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:02.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.639+0000 7fd104c5f640 1 -- 172.21.15.71:0/347379704 shutdown_connections 2024-03-29T11:10:02.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.639+0000 7fd104c5f640 1 -- 172.21.15.71:0/347379704 >> 172.21.15.71:0/347379704 conn(0x7fd10006b660 msgr2=0x7fd1000a9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:02.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.640+0000 7fd104c5f640 1 -- 172.21.15.71:0/347379704 shutdown_connections 2024-03-29T11:10:02.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.640+0000 7fd104c5f640 1 -- 172.21.15.71:0/347379704 wait complete. 2024-03-29T11:10:02.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:02.640+0000 7fd104c5f640 1 librados: shutdown 2024-03-29T11:10:02.634 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:10:09.442 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:10:09.615 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.637+0000 7efdd4bcf640 10 monclient: get_monmap_and_config 2024-03-29T11:10:09.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.637+0000 7efdd4bcf640 10 monclient: build_initial_monmap 2024-03-29T11:10:09.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.637+0000 7efdd4bcf640 10 monclient: monmap: 2024-03-29T11:10:09.616 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:10:09.616 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:09.616 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:10:09.638156+0000 2024-03-29T11:10:09.616 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:10:09.638156+0000 2024-03-29T11:10:09.616 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:09.616 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:09.616 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:10:09.616 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:10:09.616 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:10:09.616 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:09.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.638+0000 7efdd4bcf640 1 Processor -- start 2024-03-29T11:10:09.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.638+0000 7efdd4bcf640 1 -- start start 2024-03-29T11:10:09.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.638+0000 7efdd4bcf640 10 monclient: init 2024-03-29T11:10:09.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.639+0000 7efdd4bcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:09.617 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.639+0000 7efdd4bcf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:10:09.618 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.639+0000 7efdd4bcf640 10 monclient(hunting): picked mon.noname-b con 0x7efdd01422b0 addr v1:172.21.15.71:6790/0 2024-03-29T11:10:09.618 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.639+0000 7efdd4bcf640 10 monclient(hunting): picked mon.noname-a con 0x7efdd0142690 addr v1:172.21.15.71:6789/0 2024-03-29T11:10:09.618 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.639+0000 7efdd4bcf640 10 monclient(hunting): picked mon.noname-c con 0x7efdd00b17d0 addr v1:172.21.15.137:6789/0 2024-03-29T11:10:09.618 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.639+0000 7efdd4bcf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd0142c80 con 0x7efdd0142690 2024-03-29T11:10:09.618 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.639+0000 7efdd4bcf640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd00aae40 con 0x7efdd00b17d0 2024-03-29T11:10:09.618 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.639+0000 7efdd4bcf640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd013fbd0 con 0x7efdd01422b0 2024-03-29T11:10:09.618 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.639+0000 7efdd4bcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:09.618 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.639+0000 7efdd4bcf640 10 monclient(hunting): authenticate will time out at 1365.897610s 2024-03-29T11:10:09.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.643+0000 7efdce59c640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7efdd01422b0 0x7efdd013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:41978/0 (socket says 172.21.15.71:41978) 2024-03-29T11:10:09.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.643+0000 7efdce59c640 1 -- 172.21.15.71:0/964830990 learned_addr learned my addr 172.21.15.71:0/964830990 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:10:09.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.644+0000 7efdcd59a640 1 -- 172.21.15.71:0/964830990 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 117467838 0 0) 0x7efdd0142c80 con 0x7efdd0142690 2024-03-29T11:10:09.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.644+0000 7efdcd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:10:09.624 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.644+0000 7efdcd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:09.624 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.644+0000 7efdcd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:09.624 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.644+0000 7efdcd59a640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:10:09.624 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.644+0000 7efdcd59a640 1 -- 172.21.15.71:0/964830990 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdb8003520 con 0x7efdd0142690 2024-03-29T11:10:09.624 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.645+0000 7efdcd59a640 1 -- 172.21.15.71:0/964830990 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 407465903 0 0) 0x7efdb8003520 con 0x7efdd0142690 2024-03-29T11:10:09.624 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.645+0000 7efdcd59a640 1 -- 172.21.15.71:0/964830990 >> v1:172.21.15.71:6790/0 conn(0x7efdd01422b0 legacy=0x7efdd013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:09.624 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.645+0000 7efdcd59a640 1 -- 172.21.15.71:0/964830990 >> v1:172.21.15.137:6789/0 conn(0x7efdd00b17d0 legacy=0x7efdd00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:10:09.624 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.645+0000 7efdcd59a640 10 monclient: _finish_hunting 0 2024-03-29T11:10:09.624 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.645+0000 7efdcd59a640 1 monclient: found mon.noname-a 2024-03-29T11:10:09.624 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.645+0000 7efdcd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:09.624 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.645+0000 7efdcd59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:10:09.624 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.645+0000 7efdcd59a640 1 -- 172.21.15.71:0/964830990 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd0151940 con 0x7efdd0142690 2024-03-29T11:10:09.625 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.645+0000 7efdcd59a640 10 monclient: _finish_auth 0 2024-03-29T11:10:09.625 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.645+0000 7efdcd59a640 10 monclient: _check_auth_tickets 2024-03-29T11:10:09.625 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.645+0000 7efdcd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:09.625 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.645+0000 7efdcd59a640 1 -- 172.21.15.71:0/964830990 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7efdc8001940 con 0x7efdd0142690 2024-03-29T11:10:09.625 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.645+0000 7efdcd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:09.625 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.645+0000 7efdcd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:10:09.625 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.645+0000 7efdcd59a640 10 monclient: dump: 2024-03-29T11:10:09.625 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:09.625 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:09.625 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:09.626 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:09.626 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:09.626 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:09.626 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:09.626 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:09.626 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:09.626 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:09.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.645+0000 7efdd4bcf640 5 monclient: authenticate success, global_id 4584 2024-03-29T11:10:09.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.645+0000 7efdd4bcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:09.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.646+0000 7efdcd59a640 1 -- 172.21.15.71:0/964830990 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efdc8001b60 con 0x7efdd0142690 2024-03-29T11:10:09.627 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.647+0000 7efdcd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:09.627 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.647+0000 7efdcd59a640 1 -- 172.21.15.71:0/964830990 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7efdc8001ef0 con 0x7efdd0142690 2024-03-29T11:10:09.627 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.647+0000 7efdcd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:09.627 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.648+0000 7efdcd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:10:09.627 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.648+0000 7efdcd59a640 10 monclient: dump: 2024-03-29T11:10:09.627 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:09.627 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:09.627 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:09.627 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:09.627 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:09.627 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:09.628 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:09.628 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:09.628 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:09.628 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:09.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.649+0000 7efdd4bcf640 10 monclient: get_monmap_and_config success 2024-03-29T11:10:09.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.649+0000 7efdd4bcf640 4 set_mon_vals no callback set 2024-03-29T11:10:09.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.650+0000 7efdd4bcf640 10 monclient: shutdown 2024-03-29T11:10:09.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.650+0000 7efdd4bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:09.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.650+0000 7efdd4bcf640 1 -- 172.21.15.71:0/964830990 >> v1:172.21.15.71:6789/0 conn(0x7efdd0142690 legacy=0x7efdd00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:09.630 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.651+0000 7efdd4bcf640 1 -- 172.21.15.71:0/964830990 shutdown_connections 2024-03-29T11:10:09.630 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.651+0000 7efdd4bcf640 1 -- 172.21.15.71:0/964830990 >> 172.21.15.71:0/964830990 conn(0x7efdd0066a00 msgr2=0x7efdd0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:09.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.654+0000 7efdd4bcf640 1 -- 172.21.15.71:0/964830990 shutdown_connections 2024-03-29T11:10:09.633 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.654+0000 7efdd4bcf640 1 -- 172.21.15.71:0/964830990 wait complete. 2024-03-29T11:10:09.634 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.655+0000 7efdd4bcf640 10 monclient: build_initial_monmap 2024-03-29T11:10:09.634 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.655+0000 7efdd4bcf640 10 monclient: monmap: 2024-03-29T11:10:09.634 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:10:09.634 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:10:09.634 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:10:09.634 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:10:09.634 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:09.634 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:09.634 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:10:09.634 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:10:09.634 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:10:09.634 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:09.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.655+0000 7efdd4bcf640 1 librados: starting msgr at 2024-03-29T11:10:09.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.655+0000 7efdd4bcf640 1 librados: starting objecter 2024-03-29T11:10:09.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.656+0000 7efdd4bcf640 1 Processor -- start 2024-03-29T11:10:09.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.656+0000 7efdd4bcf640 1 -- start start 2024-03-29T11:10:09.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.656+0000 7efdd4bcf640 1 librados: setting wanted keys 2024-03-29T11:10:09.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.656+0000 7efdd4bcf640 1 librados: calling monclient init 2024-03-29T11:10:09.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.656+0000 7efdd4bcf640 10 monclient: init 2024-03-29T11:10:09.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.656+0000 7efdd4bcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:09.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.656+0000 7efdd4bcf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:10:09.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.656+0000 7efdd4bcf640 10 monclient(hunting): picked mon.noname-c con 0x7efdd00b17d0 addr v1:172.21.15.71:6790/0 2024-03-29T11:10:09.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.657+0000 7efdd4bcf640 10 monclient(hunting): picked mon.noname-a con 0x7efdd01422b0 addr v1:172.21.15.71:6789/0 2024-03-29T11:10:09.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.657+0000 7efdd4bcf640 10 monclient(hunting): picked mon.noname-b con 0x7efdd00b6cb0 addr v1:172.21.15.137:6789/0 2024-03-29T11:10:09.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.657+0000 7efdd4bcf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd0131b00 con 0x7efdd01422b0 2024-03-29T11:10:09.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.657+0000 7efdd4bcf640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd01326c0 con 0x7efdd00b6cb0 2024-03-29T11:10:09.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.657+0000 7efdd4bcf640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd012f680 con 0x7efdd00b17d0 2024-03-29T11:10:09.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.657+0000 7efdd4bcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:09.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.657+0000 7efdd4bcf640 10 monclient(hunting): authenticate will time out at 1365.915171s 2024-03-29T11:10:09.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.659+0000 7efdced9d640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7efdd00b6cb0 0x7efdd012ef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:39287/0 (socket says 172.21.15.71:39287) 2024-03-29T11:10:09.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.659+0000 7efdced9d640 1 -- 172.21.15.71:0/515038937 learned_addr learned my addr 172.21.15.71:0/515038937 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:10:09.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.659+0000 7efdbeffd640 10 client.?.objecter ms_handle_connect 0x7efdd00b6cb0 2024-03-29T11:10:09.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.660+0000 7efdbeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:09.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.660+0000 7efdbeffd640 10 client.?.objecter ms_handle_connect 0x7efdd00b17d0 2024-03-29T11:10:09.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.660+0000 7efdbeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:09.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.660+0000 7efdbeffd640 1 -- 172.21.15.71:0/515038937 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454616296 0 0) 0x7efdd012f680 con 0x7efdd00b17d0 2024-03-29T11:10:09.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.660+0000 7efdbeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:10:09.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.660+0000 7efdbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:09.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.660+0000 7efdbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:09.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.660+0000 7efdbeffd640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:10:09.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.660+0000 7efdbeffd640 1 -- 172.21.15.71:0/515038937 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdb0004eb0 con 0x7efdd00b17d0 2024-03-29T11:10:09.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.660+0000 7efdbeffd640 1 -- 172.21.15.71:0/515038937 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693367060 0 0) 0x7efdd01326c0 con 0x7efdd00b6cb0 2024-03-29T11:10:09.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.660+0000 7efdbeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:10:09.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.660+0000 7efdbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:09.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.660+0000 7efdbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:09.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.660+0000 7efdbeffd640 10 monclient(hunting): my global_id is 4510 2024-03-29T11:10:09.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.660+0000 7efdbeffd640 1 -- 172.21.15.71:0/515038937 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdd012f680 con 0x7efdd00b6cb0 2024-03-29T11:10:09.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.660+0000 7efdbeffd640 1 -- 172.21.15.71:0/515038937 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1480571293 0 0) 0x7efdb0004eb0 con 0x7efdd00b17d0 2024-03-29T11:10:09.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.660+0000 7efdbeffd640 1 -- 172.21.15.71:0/515038937 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efdd01326c0 con 0x7efdd00b17d0 2024-03-29T11:10:09.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.660+0000 7efdbeffd640 1 -- 172.21.15.71:0/515038937 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7efdc0001810 con 0x7efdd00b17d0 2024-03-29T11:10:09.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.660+0000 7efdbeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:09.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.660+0000 7efdbeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:10:09.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.660+0000 7efdbeffd640 10 monclient(hunting): dump: 2024-03-29T11:10:09.642 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:09.642 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:09.642 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:09.642 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:09.642 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:09.642 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:09.642 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:09.642 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:09.642 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:09.642 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:09.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.660+0000 7efdbeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:10:09.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.660+0000 7efdbeffd640 1 -- 172.21.15.71:0/515038937 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3548812509 0 0) 0x7efdd012f680 con 0x7efdd00b6cb0 2024-03-29T11:10:09.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.661+0000 7efdbeffd640 1 -- 172.21.15.71:0/515038937 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efdb0004eb0 con 0x7efdd00b6cb0 2024-03-29T11:10:09.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.661+0000 7efdbeffd640 1 -- 172.21.15.71:0/515038937 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4087845207 0 0) 0x7efdd01326c0 con 0x7efdd00b17d0 2024-03-29T11:10:09.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.661+0000 7efdbeffd640 1 -- 172.21.15.71:0/515038937 >> v1:172.21.15.137:6789/0 conn(0x7efdd00b6cb0 legacy=0x7efdd012ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:09.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.661+0000 7efdbeffd640 1 -- 172.21.15.71:0/515038937 >> v1:172.21.15.71:6789/0 conn(0x7efdd01422b0 legacy=0x7efdd012e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:09.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.661+0000 7efdbeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:10:09.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.661+0000 7efdbeffd640 1 monclient: found mon.c 2024-03-29T11:10:09.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.661+0000 7efdbeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:09.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.661+0000 7efdbeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:10:09.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.661+0000 7efdbeffd640 1 -- 172.21.15.71:0/515038937 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd006ecd0 con 0x7efdd00b17d0 2024-03-29T11:10:09.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.661+0000 7efdbeffd640 10 monclient: _finish_auth 0 2024-03-29T11:10:09.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.661+0000 7efdbeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:10:09.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.661+0000 7efdbeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:09.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.661+0000 7efdd4bcf640 5 monclient: authenticate success, global_id 4643 2024-03-29T11:10:09.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.661+0000 7efdd4bcf640 10 monclient: _renew_subs 2024-03-29T11:10:09.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.661+0000 7efdd4bcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:10:09.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.661+0000 7efdd4bcf640 1 -- 172.21.15.71:0/515038937 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdd006efb0 con 0x7efdd00b17d0 2024-03-29T11:10:09.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.661+0000 7efdd4bcf640 10 client.4643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:10:09.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.661+0000 7efdd4bcf640 10 monclient: _renew_subs 2024-03-29T11:10:09.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.661+0000 7efdd4bcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:10:09.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.661+0000 7efdd4bcf640 1 -- 172.21.15.71:0/515038937 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efdd012fc60 con 0x7efdd00b17d0 2024-03-29T11:10:09.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.661+0000 7efdd4bcf640 1 librados: init done 2024-03-29T11:10:09.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.663+0000 7efdd4bcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:09.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.663+0000 7efdd4bcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:09.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.663+0000 7efdd4bcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:10:09.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.663+0000 7efdd4bcf640 1 -- 172.21.15.71:0/515038937 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdd0075f10 con 0x7efdd00b17d0 2024-03-29T11:10:09.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.664+0000 7efdbeffd640 1 -- 172.21.15.71:0/515038937 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efdc0001d00 con 0x7efdd00b17d0 2024-03-29T11:10:09.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.664+0000 7efdbeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:09.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.664+0000 7efdbffff640 4 set_mon_vals no callback set 2024-03-29T11:10:09.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.665+0000 7efdbeffd640 1 -- 172.21.15.71:0/515038937 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7efdc0004980 con 0x7efdd00b17d0 2024-03-29T11:10:09.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.665+0000 7efdbeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:09.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.665+0000 7efdbeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:10:09.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.665+0000 7efdbeffd640 10 monclient: dump: 2024-03-29T11:10:09.646 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:09.646 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:09.646 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:09.646 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:09.646 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:09.646 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:09.646 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:09.646 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:09.646 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:09.646 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:09.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.669+0000 7efdbeffd640 1 -- 172.21.15.71:0/515038937 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7efdc0004ae0 con 0x7efdd00b17d0 2024-03-29T11:10:09.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.670+0000 7efdcdd9b640 10 monclient: get_auth_request con 0x7efdb006dd00 auth_method 0 2024-03-29T11:10:09.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.670+0000 7efdbeffd640 1 -- 172.21.15.71:0/515038937 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7efdc0084890 con 0x7efdd00b17d0 2024-03-29T11:10:09.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.670+0000 7efdbeffd640 10 client.4643.objecter ms_dispatch 0x7efdd0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:10:09.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.670+0000 7efdbeffd640 3 client.4643.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:10:09.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.670+0000 7efdbeffd640 3 client.4643.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:10:09.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.670+0000 7efdbeffd640 20 client.4643.objecter dump_active .. 0 homeless 2024-03-29T11:10:09.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.673+0000 7efdbeffd640 10 client.4643.objecter ms_handle_connect 0x7efdb006dd00 2024-03-29T11:10:09.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.674+0000 7efdbeffd640 1 -- 172.21.15.71:0/515038937 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efdc0050170 con 0x7efdd00b17d0 2024-03-29T11:10:09.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.674+0000 7efdbeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:09.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:09.674+0000 7efdbeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:10:10.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:10.030+0000 7efdd4bcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:10.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:10.030+0000 7efdd4bcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:10.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:10.030+0000 7efdd4bcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:10:10.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:10.030+0000 7efdd4bcf640 1 -- 172.21.15.71:0/515038937 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efdd00b2270 con 0x7efdd00b17d0 2024-03-29T11:10:10.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:10.031+0000 7efdbeffd640 1 -- 172.21.15.71:0/515038937 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7efdc00520a0 con 0x7efdd00b17d0 2024-03-29T11:10:10.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:10.031+0000 7efdbeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:10.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:10.031+0000 7efdbeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:10:10.010 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:10:10.010 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:10:10.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:10.035+0000 7efdd4bcf640 10 librados: watch_flush enter 2024-03-29T11:10:10.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:10.035+0000 7efdd4bcf640 10 librados: watch_flush exit 2024-03-29T11:10:10.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:10.035+0000 7efdd4bcf640 20 client.4643.objecter shutdown clearing up homeless session... 2024-03-29T11:10:10.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:10.035+0000 7efdd4bcf640 10 client.4643.objecter successfully canceled tick 2024-03-29T11:10:10.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:10.036+0000 7efdd4bcf640 1 -- 172.21.15.71:0/515038937 >> v1:172.21.15.137:6816/973332665 conn(0x7efdb006dd00 legacy=0x7efdb00701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:10.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:10.036+0000 7efdd4bcf640 10 monclient: shutdown 2024-03-29T11:10:10.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:10.036+0000 7efdd4bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:10.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:10.036+0000 7efdd4bcf640 1 -- 172.21.15.71:0/515038937 >> v1:172.21.15.71:6790/0 conn(0x7efdd00b17d0 legacy=0x7efdd0131fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:10.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:10.036+0000 7efdd4bcf640 1 -- 172.21.15.71:0/515038937 shutdown_connections 2024-03-29T11:10:10.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:10.036+0000 7efdd4bcf640 1 -- 172.21.15.71:0/515038937 >> 172.21.15.71:0/515038937 conn(0x7efdd0066a00 msgr2=0x7efdd013d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:10.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:10.036+0000 7efdd4bcf640 1 -- 172.21.15.71:0/515038937 shutdown_connections 2024-03-29T11:10:10.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:10.036+0000 7efdd4bcf640 1 -- 172.21.15.71:0/515038937 wait complete. 2024-03-29T11:10:10.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:10.036+0000 7efdd4bcf640 1 librados: shutdown 2024-03-29T11:10:10.030 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:10:16.837 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:10:17.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.027+0000 7fcd4bd9d640 10 monclient: get_monmap_and_config 2024-03-29T11:10:17.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.027+0000 7fcd4bd9d640 10 monclient: build_initial_monmap 2024-03-29T11:10:17.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.027+0000 7fcd4bd9d640 10 monclient: monmap: 2024-03-29T11:10:17.007 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:10:17.007 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:17.007 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:10:17.028331+0000 2024-03-29T11:10:17.007 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:10:17.028331+0000 2024-03-29T11:10:17.007 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:17.007 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:17.007 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:10:17.007 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:10:17.007 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:10:17.007 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:17.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.029+0000 7fcd4bd9d640 1 Processor -- start 2024-03-29T11:10:17.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.029+0000 7fcd4bd9d640 1 -- start start 2024-03-29T11:10:17.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.029+0000 7fcd4bd9d640 10 monclient: init 2024-03-29T11:10:17.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.030+0000 7fcd4bd9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:17.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.030+0000 7fcd4bd9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:10:17.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.030+0000 7fcd4bd9d640 10 monclient(hunting): picked mon.noname-c con 0x7fcd440a9ac0 addr v1:172.21.15.137:6789/0 2024-03-29T11:10:17.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.030+0000 7fcd4bd9d640 10 monclient(hunting): picked mon.noname-b con 0x7fcd4413ee90 addr v1:172.21.15.71:6790/0 2024-03-29T11:10:17.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.030+0000 7fcd4bd9d640 10 monclient(hunting): picked mon.noname-a con 0x7fcd440af900 addr v1:172.21.15.71:6789/0 2024-03-29T11:10:17.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.030+0000 7fcd4bd9d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd440afd10 con 0x7fcd440af900 2024-03-29T11:10:17.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.030+0000 7fcd4bd9d640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd440af6f0 con 0x7fcd440a9ac0 2024-03-29T11:10:17.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.031+0000 7fcd4bd9d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd440aac80 con 0x7fcd4413ee90 2024-03-29T11:10:17.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.031+0000 7fcd4bd9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:17.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.031+0000 7fcd4bd9d640 10 monclient(hunting): authenticate will time out at 1373.289124s 2024-03-29T11:10:17.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.034+0000 7fcd49b39640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fcd440a9ac0 0x7fcd440adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:44289/0 (socket says 172.21.15.71:44289) 2024-03-29T11:10:17.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.034+0000 7fcd49b39640 1 -- 172.21.15.71:0/880077461 learned_addr learned my addr 172.21.15.71:0/880077461 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:10:17.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.035+0000 7fcd48b37640 1 -- 172.21.15.71:0/880077461 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1764272663 0 0) 0x7fcd440afd10 con 0x7fcd440af900 2024-03-29T11:10:17.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.035+0000 7fcd48b37640 10 monclient(hunting): _negotiate 2024-03-29T11:10:17.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.035+0000 7fcd48b37640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:17.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.035+0000 7fcd48b37640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:17.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.035+0000 7fcd48b37640 10 monclient(hunting): my global_id is 4587 2024-03-29T11:10:17.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.035+0000 7fcd48b37640 1 -- 172.21.15.71:0/880077461 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd30003520 con 0x7fcd440af900 2024-03-29T11:10:17.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.035+0000 7fcd48b37640 1 -- 172.21.15.71:0/880077461 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4265056298 0 0) 0x7fcd440af6f0 con 0x7fcd440a9ac0 2024-03-29T11:10:17.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.035+0000 7fcd48b37640 10 monclient(hunting): _negotiate 2024-03-29T11:10:17.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.035+0000 7fcd48b37640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:17.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.035+0000 7fcd48b37640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:17.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.035+0000 7fcd48b37640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:10:17.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.035+0000 7fcd48b37640 1 -- 172.21.15.71:0/880077461 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd440afd10 con 0x7fcd440a9ac0 2024-03-29T11:10:17.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.035+0000 7fcd48b37640 1 -- 172.21.15.71:0/880077461 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3355490637 0 0) 0x7fcd440aac80 con 0x7fcd4413ee90 2024-03-29T11:10:17.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.035+0000 7fcd48b37640 10 monclient(hunting): _negotiate 2024-03-29T11:10:17.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.035+0000 7fcd48b37640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:17.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.035+0000 7fcd48b37640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:17.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.035+0000 7fcd48b37640 10 monclient(hunting): my global_id is 4646 2024-03-29T11:10:17.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.035+0000 7fcd48b37640 1 -- 172.21.15.71:0/880077461 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd440af6f0 con 0x7fcd4413ee90 2024-03-29T11:10:17.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.036+0000 7fcd48b37640 1 -- 172.21.15.71:0/880077461 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 126217349 0 0) 0x7fcd440afd10 con 0x7fcd440a9ac0 2024-03-29T11:10:17.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.036+0000 7fcd48b37640 1 -- 172.21.15.71:0/880077461 >> v1:172.21.15.71:6790/0 conn(0x7fcd4413ee90 legacy=0x7fcd4413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:17.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.036+0000 7fcd48b37640 1 -- 172.21.15.71:0/880077461 >> v1:172.21.15.71:6789/0 conn(0x7fcd440af900 legacy=0x7fcd44149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:17.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.036+0000 7fcd48b37640 10 monclient: _finish_hunting 0 2024-03-29T11:10:17.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.036+0000 7fcd48b37640 1 monclient: found mon.noname-c 2024-03-29T11:10:17.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.036+0000 7fcd48b37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:17.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.036+0000 7fcd48b37640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.137:6789/0 2024-03-29T11:10:17.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.036+0000 7fcd48b37640 1 -- 172.21.15.71:0/880077461 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd440aaf60 con 0x7fcd440a9ac0 2024-03-29T11:10:17.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.036+0000 7fcd48b37640 10 monclient: _finish_auth 0 2024-03-29T11:10:17.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.036+0000 7fcd48b37640 10 monclient: _check_auth_tickets 2024-03-29T11:10:17.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.036+0000 7fcd48b37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:17.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.036+0000 7fcd48b37640 1 -- 172.21.15.71:0/880077461 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fcd3c005010 con 0x7fcd440a9ac0 2024-03-29T11:10:17.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.036+0000 7fcd48b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:17.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.036+0000 7fcd48b37640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:10:17.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.036+0000 7fcd48b37640 10 monclient: dump: 2024-03-29T11:10:17.017 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:17.018 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:17.018 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:17.018 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:17.018 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:17.018 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:17.018 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:17.018 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:17.018 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:17.018 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:17.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.036+0000 7fcd4bd9d640 5 monclient: authenticate success, global_id 4513 2024-03-29T11:10:17.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.036+0000 7fcd4bd9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:17.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.036+0000 7fcd48b37640 1 -- 172.21.15.71:0/880077461 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd3c005170 con 0x7fcd440a9ac0 2024-03-29T11:10:17.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.036+0000 7fcd48b37640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:17.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.036+0000 7fcd48b37640 1 -- 172.21.15.71:0/880077461 <== mon.1 v1:172.21.15.137:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fcd3c001770 con 0x7fcd440a9ac0 2024-03-29T11:10:17.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.036+0000 7fcd48b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:17.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.036+0000 7fcd48b37640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:10:17.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.036+0000 7fcd48b37640 10 monclient: dump: 2024-03-29T11:10:17.019 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:17.019 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:17.019 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:17.019 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:17.019 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:17.019 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:17.019 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:17.019 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:17.019 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:17.019 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:17.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.036+0000 7fcd4bd9d640 10 monclient: get_monmap_and_config success 2024-03-29T11:10:17.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.036+0000 7fcd4bd9d640 4 set_mon_vals no callback set 2024-03-29T11:10:17.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.037+0000 7fcd4bd9d640 10 monclient: shutdown 2024-03-29T11:10:17.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.037+0000 7fcd4bd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:17.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.037+0000 7fcd4bd9d640 1 -- 172.21.15.71:0/880077461 >> v1:172.21.15.137:6789/0 conn(0x7fcd440a9ac0 legacy=0x7fcd440adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:17.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.038+0000 7fcd4bd9d640 1 -- 172.21.15.71:0/880077461 shutdown_connections 2024-03-29T11:10:17.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.038+0000 7fcd4bd9d640 1 -- 172.21.15.71:0/880077461 >> 172.21.15.71:0/880077461 conn(0x7fcd440b35e0 msgr2=0x7fcd440b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:17.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.039+0000 7fcd4bd9d640 1 -- 172.21.15.71:0/880077461 shutdown_connections 2024-03-29T11:10:17.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.039+0000 7fcd4bd9d640 1 -- 172.21.15.71:0/880077461 wait complete. 2024-03-29T11:10:17.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.040+0000 7fcd4bd9d640 10 monclient: build_initial_monmap 2024-03-29T11:10:17.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.040+0000 7fcd4bd9d640 10 monclient: monmap: 2024-03-29T11:10:17.021 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:10:17.021 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:10:17.021 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:10:17.021 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:10:17.021 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:17.021 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:17.021 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:10:17.022 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:10:17.022 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:10:17.022 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:17.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.041+0000 7fcd4bd9d640 1 librados: starting msgr at 2024-03-29T11:10:17.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.041+0000 7fcd4bd9d640 1 librados: starting objecter 2024-03-29T11:10:17.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.041+0000 7fcd4bd9d640 1 Processor -- start 2024-03-29T11:10:17.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.043+0000 7fcd4bd9d640 1 -- start start 2024-03-29T11:10:17.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.043+0000 7fcd4bd9d640 1 librados: setting wanted keys 2024-03-29T11:10:17.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.043+0000 7fcd4bd9d640 1 librados: calling monclient init 2024-03-29T11:10:17.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.043+0000 7fcd4bd9d640 10 monclient: init 2024-03-29T11:10:17.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.044+0000 7fcd4bd9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:17.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.044+0000 7fcd4bd9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:10:17.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.044+0000 7fcd4bd9d640 10 monclient(hunting): picked mon.noname-c con 0x7fcd440af900 addr v1:172.21.15.71:6790/0 2024-03-29T11:10:17.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.044+0000 7fcd4bd9d640 10 monclient(hunting): picked mon.noname-a con 0x7fcd4413ee90 addr v1:172.21.15.71:6789/0 2024-03-29T11:10:17.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.044+0000 7fcd4bd9d640 10 monclient(hunting): picked mon.noname-b con 0x7fcd441dfe80 addr v1:172.21.15.137:6789/0 2024-03-29T11:10:17.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.044+0000 7fcd4bd9d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd441c8ab0 con 0x7fcd4413ee90 2024-03-29T11:10:17.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.044+0000 7fcd4bd9d640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd441dbcf0 con 0x7fcd441dfe80 2024-03-29T11:10:17.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.044+0000 7fcd4bd9d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd441dbe80 con 0x7fcd440af900 2024-03-29T11:10:17.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.044+0000 7fcd4bd9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:17.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.044+0000 7fcd4bd9d640 10 monclient(hunting): authenticate will time out at 1373.302493s 2024-03-29T11:10:17.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.049+0000 7fcd49338640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fcd4413ee90 0x7fcd44138b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:48007/0 (socket says 172.21.15.71:48007) 2024-03-29T11:10:17.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.049+0000 7fcd49338640 1 -- 172.21.15.71:0/2070777997 learned_addr learned my addr 172.21.15.71:0/2070777997 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:10:17.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.049+0000 7fcd49b39640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fcd440af900 0x7fcd441df770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:50050/0 (socket says 172.21.15.71:50050) 2024-03-29T11:10:17.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.052+0000 7fcd3a7fc640 10 client.?.objecter ms_handle_connect 0x7fcd440af900 2024-03-29T11:10:17.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.052+0000 7fcd3a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:17.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.052+0000 7fcd3a7fc640 10 client.?.objecter ms_handle_connect 0x7fcd441dfe80 2024-03-29T11:10:17.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.052+0000 7fcd3a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:17.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.052+0000 7fcd3a7fc640 10 client.?.objecter ms_handle_connect 0x7fcd4413ee90 2024-03-29T11:10:17.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.052+0000 7fcd3a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:17.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.052+0000 7fcd3a7fc640 1 -- 172.21.15.71:0/2070777997 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1435052603 0 0) 0x7fcd441dbe80 con 0x7fcd440af900 2024-03-29T11:10:17.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.052+0000 7fcd3a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:10:17.030 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.052+0000 7fcd3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:17.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.052+0000 7fcd3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:17.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.052+0000 7fcd3a7fc640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:10:17.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.052+0000 7fcd3a7fc640 1 -- 172.21.15.71:0/2070777997 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd28004eb0 con 0x7fcd440af900 2024-03-29T11:10:17.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.054+0000 7fcd3a7fc640 1 -- 172.21.15.71:0/2070777997 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1460593289 0 0) 0x7fcd441c8ab0 con 0x7fcd4413ee90 2024-03-29T11:10:17.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.054+0000 7fcd3a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:10:17.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.054+0000 7fcd3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:17.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.054+0000 7fcd3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:17.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.054+0000 7fcd3a7fc640 10 monclient(hunting): my global_id is 4590 2024-03-29T11:10:17.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.054+0000 7fcd3a7fc640 1 -- 172.21.15.71:0/2070777997 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd441dbe80 con 0x7fcd4413ee90 2024-03-29T11:10:17.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.054+0000 7fcd3a7fc640 1 -- 172.21.15.71:0/2070777997 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441527850 0 0) 0x7fcd441dbcf0 con 0x7fcd441dfe80 2024-03-29T11:10:17.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.054+0000 7fcd3a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:10:17.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.054+0000 7fcd3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:17.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.054+0000 7fcd3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:17.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.054+0000 7fcd3a7fc640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:10:17.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.054+0000 7fcd3a7fc640 1 -- 172.21.15.71:0/2070777997 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd441c8ab0 con 0x7fcd441dfe80 2024-03-29T11:10:17.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.054+0000 7fcd3a7fc640 1 -- 172.21.15.71:0/2070777997 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4282530517 0 0) 0x7fcd28004eb0 con 0x7fcd440af900 2024-03-29T11:10:17.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.054+0000 7fcd3a7fc640 1 -- 172.21.15.71:0/2070777997 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd441dbcf0 con 0x7fcd440af900 2024-03-29T11:10:17.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.054+0000 7fcd3a7fc640 1 -- 172.21.15.71:0/2070777997 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fcd3c001730 con 0x7fcd440af900 2024-03-29T11:10:17.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.054+0000 7fcd3a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:17.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.054+0000 7fcd3a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:10:17.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.054+0000 7fcd3a7fc640 10 monclient(hunting): dump: 2024-03-29T11:10:17.038 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:17.038 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:17.038 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:17.038 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:17.038 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:17.038 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:17.038 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:17.039 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:17.039 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:17.039 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:17.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.054+0000 7fcd3a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:10:17.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.054+0000 7fcd3a7fc640 1 -- 172.21.15.71:0/2070777997 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2234175907 0 0) 0x7fcd441c8ab0 con 0x7fcd441dfe80 2024-03-29T11:10:17.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.054+0000 7fcd3a7fc640 1 -- 172.21.15.71:0/2070777997 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd28004eb0 con 0x7fcd441dfe80 2024-03-29T11:10:17.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.055+0000 7fcd3a7fc640 1 -- 172.21.15.71:0/2070777997 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fcd400018d0 con 0x7fcd441dfe80 2024-03-29T11:10:17.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.055+0000 7fcd3a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:17.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.055+0000 7fcd3a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:10:17.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.055+0000 7fcd3a7fc640 10 monclient(hunting): dump: 2024-03-29T11:10:17.039 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:17.039 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:17.039 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:17.039 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:17.040 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:17.040 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:17.040 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:17.040 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:17.040 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:17.040 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:17.040 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.055+0000 7fcd3a7fc640 1 -- 172.21.15.71:0/2070777997 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2701579007 0 0) 0x7fcd441dbe80 con 0x7fcd4413ee90 2024-03-29T11:10:17.040 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.055+0000 7fcd3a7fc640 1 -- 172.21.15.71:0/2070777997 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd441c8ab0 con 0x7fcd4413ee90 2024-03-29T11:10:17.040 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.055+0000 7fcd3a7fc640 1 -- 172.21.15.71:0/2070777997 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fcd340018d0 con 0x7fcd4413ee90 2024-03-29T11:10:17.040 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.055+0000 7fcd3a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:17.040 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.055+0000 7fcd3a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:10:17.040 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.055+0000 7fcd3a7fc640 10 monclient(hunting): dump: 2024-03-29T11:10:17.040 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:17.041 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:17.041 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:17.041 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:17.041 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:17.041 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:17.041 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:17.041 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:17.041 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:17.041 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:17.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.055+0000 7fcd3a7fc640 1 -- 172.21.15.71:0/2070777997 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1565865994 0 0) 0x7fcd28004eb0 con 0x7fcd441dfe80 2024-03-29T11:10:17.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.056+0000 7fcd3a7fc640 1 -- 172.21.15.71:0/2070777997 >> v1:172.21.15.71:6790/0 conn(0x7fcd440af900 legacy=0x7fcd441df770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:17.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.056+0000 7fcd3a7fc640 1 -- 172.21.15.71:0/2070777997 >> v1:172.21.15.71:6789/0 conn(0x7fcd4413ee90 legacy=0x7fcd44138b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:17.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.056+0000 7fcd3a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:10:17.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.056+0000 7fcd3a7fc640 1 monclient: found mon.b 2024-03-29T11:10:17.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.056+0000 7fcd3a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:17.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.056+0000 7fcd3a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:10:17.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.056+0000 7fcd3a7fc640 1 -- 172.21.15.71:0/2070777997 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd44139230 con 0x7fcd441dfe80 2024-03-29T11:10:17.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.056+0000 7fcd3a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:10:17.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.056+0000 7fcd3a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:10:17.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.056+0000 7fcd3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:17.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.056+0000 7fcd4bd9d640 5 monclient: authenticate success, global_id 4516 2024-03-29T11:10:17.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.056+0000 7fcd4bd9d640 10 monclient: _renew_subs 2024-03-29T11:10:17.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.056+0000 7fcd4bd9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:10:17.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.056+0000 7fcd4bd9d640 1 -- 172.21.15.71:0/2070777997 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd44139480 con 0x7fcd441dfe80 2024-03-29T11:10:17.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.056+0000 7fcd4bd9d640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:10:17.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.056+0000 7fcd4bd9d640 10 monclient: _renew_subs 2024-03-29T11:10:17.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.056+0000 7fcd4bd9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:10:17.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.056+0000 7fcd4bd9d640 1 -- 172.21.15.71:0/2070777997 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd44139dc0 con 0x7fcd441dfe80 2024-03-29T11:10:17.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.056+0000 7fcd4bd9d640 1 librados: init done 2024-03-29T11:10:17.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.056+0000 7fcd3a7fc640 1 -- 172.21.15.71:0/2070777997 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd40001ce0 con 0x7fcd441dfe80 2024-03-29T11:10:17.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.056+0000 7fcd3a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:17.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.056+0000 7fcd3a7fc640 1 -- 172.21.15.71:0/2070777997 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fcd400037b0 con 0x7fcd441dfe80 2024-03-29T11:10:17.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.056+0000 7fcd3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:17.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.056+0000 7fcd3a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:10:17.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.056+0000 7fcd3a7fc640 10 monclient: dump: 2024-03-29T11:10:17.043 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:17.043 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:17.043 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:17.043 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:17.044 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:17.044 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:17.044 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:17.044 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:17.044 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:17.044 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:17.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.056+0000 7fcd3b7fe640 4 set_mon_vals no callback set 2024-03-29T11:10:17.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.058+0000 7fcd4bd9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:17.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.058+0000 7fcd4bd9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:17.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.058+0000 7fcd4bd9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:10:17.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.058+0000 7fcd4bd9d640 1 -- 172.21.15.71:0/2070777997 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd2c005080 con 0x7fcd441dfe80 2024-03-29T11:10:17.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.060+0000 7fcd3a7fc640 1 -- 172.21.15.71:0/2070777997 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7fcd40019780 con 0x7fcd441dfe80 2024-03-29T11:10:17.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.061+0000 7fcd3a7fc640 1 -- 172.21.15.71:0/2070777997 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7fcd40083660 con 0x7fcd441dfe80 2024-03-29T11:10:17.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.061+0000 7fcd3a7fc640 10 client.4516.objecter ms_dispatch 0x7fcd44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:10:17.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.061+0000 7fcd3a7fc640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:10:17.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.061+0000 7fcd3a7fc640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:10:17.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.061+0000 7fcd3a7fc640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-29T11:10:17.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.061+0000 7fcd49b39640 10 monclient: get_auth_request con 0x7fcd2806e1d0 auth_method 0 2024-03-29T11:10:17.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.061+0000 7fcd3a7fc640 10 client.4516.objecter ms_handle_connect 0x7fcd2806e1d0 2024-03-29T11:10:17.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.066+0000 7fcd3a7fc640 1 -- 172.21.15.71:0/2070777997 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcd4004ed70 con 0x7fcd441dfe80 2024-03-29T11:10:17.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.067+0000 7fcd3a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:17.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.067+0000 7fcd3a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:10:17.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.415+0000 7fcd4bd9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:17.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.415+0000 7fcd4bd9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:17.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.415+0000 7fcd4bd9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:10:17.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.415+0000 7fcd4bd9d640 1 -- 172.21.15.71:0/2070777997 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd2c005330 con 0x7fcd441dfe80 2024-03-29T11:10:17.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.417+0000 7fcd3a7fc640 1 -- 172.21.15.71:0/2070777997 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7fcd40050e70 con 0x7fcd441dfe80 2024-03-29T11:10:17.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.417+0000 7fcd3a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:17.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.417+0000 7fcd3a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:10:17.396 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:10:17.396 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:10:17.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.421+0000 7fcd4bd9d640 10 librados: watch_flush enter 2024-03-29T11:10:17.400 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.421+0000 7fcd4bd9d640 10 librados: watch_flush exit 2024-03-29T11:10:17.400 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.422+0000 7fcd4bd9d640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-29T11:10:17.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.422+0000 7fcd4bd9d640 10 client.4516.objecter successfully canceled tick 2024-03-29T11:10:17.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.422+0000 7fcd4bd9d640 1 -- 172.21.15.71:0/2070777997 >> v1:172.21.15.137:6816/973332665 conn(0x7fcd2806e1d0 legacy=0x7fcd28070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:17.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.422+0000 7fcd4bd9d640 10 monclient: shutdown 2024-03-29T11:10:17.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.422+0000 7fcd4bd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:17.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.422+0000 7fcd4bd9d640 1 -- 172.21.15.71:0/2070777997 >> v1:172.21.15.137:6789/0 conn(0x7fcd441dfe80 legacy=0x7fcd441e7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:17.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.422+0000 7fcd4bd9d640 1 -- 172.21.15.71:0/2070777997 shutdown_connections 2024-03-29T11:10:17.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.422+0000 7fcd4bd9d640 1 -- 172.21.15.71:0/2070777997 >> 172.21.15.71:0/2070777997 conn(0x7fcd440b35e0 msgr2=0x7fcd440adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:17.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.422+0000 7fcd4bd9d640 1 -- 172.21.15.71:0/2070777997 shutdown_connections 2024-03-29T11:10:17.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.422+0000 7fcd4bd9d640 1 -- 172.21.15.71:0/2070777997 wait complete. 2024-03-29T11:10:17.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:17.423+0000 7fcd4bd9d640 1 librados: shutdown 2024-03-29T11:10:17.420 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:10:24.227 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:10:24.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.398+0000 7f26293f3640 10 monclient: get_monmap_and_config 2024-03-29T11:10:24.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.398+0000 7f26293f3640 10 monclient: build_initial_monmap 2024-03-29T11:10:24.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.398+0000 7f26293f3640 10 monclient: monmap: 2024-03-29T11:10:24.378 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:10:24.378 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:24.378 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:10:24.399507+0000 2024-03-29T11:10:24.378 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:10:24.399507+0000 2024-03-29T11:10:24.378 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:24.378 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:24.378 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:10:24.378 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:10:24.379 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:10:24.379 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:24.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.399+0000 7f26293f3640 1 Processor -- start 2024-03-29T11:10:24.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.399+0000 7f26293f3640 1 -- start start 2024-03-29T11:10:24.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.399+0000 7f26293f3640 10 monclient: init 2024-03-29T11:10:24.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.400+0000 7f26293f3640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:24.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.400+0000 7f26293f3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:10:24.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.400+0000 7f26293f3640 10 monclient(hunting): picked mon.noname-b con 0x7f26240acfa0 addr v1:172.21.15.71:6790/0 2024-03-29T11:10:24.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.400+0000 7f26293f3640 10 monclient(hunting): picked mon.noname-a con 0x7f2624072f10 addr v1:172.21.15.71:6789/0 2024-03-29T11:10:24.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.400+0000 7f26293f3640 10 monclient(hunting): picked mon.noname-c con 0x7f26240ad380 addr v1:172.21.15.137:6789/0 2024-03-29T11:10:24.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.400+0000 7f26293f3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26240aa9f0 con 0x7f2624072f10 2024-03-29T11:10:24.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.400+0000 7f26293f3640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26240ac4c0 con 0x7f26240ad380 2024-03-29T11:10:24.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.400+0000 7f26293f3640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26240ac7a0 con 0x7f26240acfa0 2024-03-29T11:10:24.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.400+0000 7f26293f3640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:24.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.400+0000 7f26293f3640 10 monclient(hunting): authenticate will time out at 1380.658994s 2024-03-29T11:10:24.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.404+0000 7f2623fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f26240acfa0 0x7f2624141450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:47228/0 (socket says 172.21.15.71:47228) 2024-03-29T11:10:24.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.404+0000 7f2623fff640 1 -- 172.21.15.71:0/4112291969 learned_addr learned my addr 172.21.15.71:0/4112291969 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:10:24.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.404+0000 7f2622ffd640 1 -- 172.21.15.71:0/4112291969 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1627719127 0 0) 0x7f26240ac7a0 con 0x7f26240acfa0 2024-03-29T11:10:24.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.404+0000 7f2622ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:10:24.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.404+0000 7f2622ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:24.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.404+0000 7f2622ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:24.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.404+0000 7f2622ffd640 10 monclient(hunting): my global_id is 4652 2024-03-29T11:10:24.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.404+0000 7f2622ffd640 1 -- 172.21.15.71:0/4112291969 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f260c003520 con 0x7f26240acfa0 2024-03-29T11:10:24.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.405+0000 7f2622ffd640 1 -- 172.21.15.71:0/4112291969 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512986837 0 0) 0x7f26240aa9f0 con 0x7f2624072f10 2024-03-29T11:10:24.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.405+0000 7f2622ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:10:24.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.405+0000 7f2622ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:24.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.405+0000 7f2622ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:24.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.405+0000 7f2622ffd640 10 monclient(hunting): my global_id is 4593 2024-03-29T11:10:24.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.405+0000 7f2622ffd640 1 -- 172.21.15.71:0/4112291969 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26240ac7a0 con 0x7f2624072f10 2024-03-29T11:10:24.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.406+0000 7f2622ffd640 1 -- 172.21.15.71:0/4112291969 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3381788374 0 0) 0x7f260c003520 con 0x7f26240acfa0 2024-03-29T11:10:24.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.406+0000 7f2622ffd640 1 -- 172.21.15.71:0/4112291969 >> v1:172.21.15.137:6789/0 conn(0x7f26240ad380 legacy=0x7f26241497f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:10:24.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.406+0000 7f2622ffd640 1 -- 172.21.15.71:0/4112291969 >> v1:172.21.15.71:6789/0 conn(0x7f2624072f10 legacy=0x7f2624141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:24.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.406+0000 7f2622ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:10:24.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.406+0000 7f2622ffd640 1 monclient: found mon.noname-b 2024-03-29T11:10:24.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.406+0000 7f2622ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:24.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.406+0000 7f2622ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:10:24.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.406+0000 7f2622ffd640 1 -- 172.21.15.71:0/4112291969 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26240ab5a0 con 0x7f26240acfa0 2024-03-29T11:10:24.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.406+0000 7f2622ffd640 10 monclient: _finish_auth 0 2024-03-29T11:10:24.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.406+0000 7f2622ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:10:24.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.406+0000 7f2622ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:24.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.406+0000 7f2622ffd640 1 -- 172.21.15.71:0/4112291969 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2614004cb0 con 0x7f26240acfa0 2024-03-29T11:10:24.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.406+0000 7f2622ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:24.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.406+0000 7f2622ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:10:24.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.406+0000 7f2622ffd640 10 monclient: dump: 2024-03-29T11:10:24.387 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:24.387 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:24.387 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:24.387 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:24.388 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:24.388 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:24.388 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:24.388 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:24.388 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:24.388 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:24.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.407+0000 7f2622ffd640 1 -- 172.21.15.71:0/4112291969 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2614004ed0 con 0x7f26240acfa0 2024-03-29T11:10:24.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.407+0000 7f2622ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:24.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.407+0000 7f2622ffd640 1 -- 172.21.15.71:0/4112291969 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f26140013f0 con 0x7f26240acfa0 2024-03-29T11:10:24.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.407+0000 7f2622ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:24.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.407+0000 7f2622ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:10:24.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.407+0000 7f2622ffd640 10 monclient: dump: 2024-03-29T11:10:24.388 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:24.389 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:24.389 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:24.389 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:24.389 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:24.389 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:24.389 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:24.389 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:24.389 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:24.389 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:24.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.408+0000 7f26293f3640 5 monclient: authenticate success, global_id 4652 2024-03-29T11:10:24.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.408+0000 7f26293f3640 10 monclient: get_monmap_and_config success 2024-03-29T11:10:24.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.408+0000 7f26293f3640 4 set_mon_vals no callback set 2024-03-29T11:10:24.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.408+0000 7f26293f3640 10 monclient: shutdown 2024-03-29T11:10:24.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.408+0000 7f26293f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:24.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.408+0000 7f26293f3640 1 -- 172.21.15.71:0/4112291969 >> v1:172.21.15.71:6790/0 conn(0x7f26240acfa0 legacy=0x7f2624141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:24.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.409+0000 7f26293f3640 1 -- 172.21.15.71:0/4112291969 shutdown_connections 2024-03-29T11:10:24.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.409+0000 7f26293f3640 1 -- 172.21.15.71:0/4112291969 >> 172.21.15.71:0/4112291969 conn(0x7f26240b3a30 msgr2=0x7f26240b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:24.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.409+0000 7f26293f3640 1 -- 172.21.15.71:0/4112291969 shutdown_connections 2024-03-29T11:10:24.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.410+0000 7f26293f3640 1 -- 172.21.15.71:0/4112291969 wait complete. 2024-03-29T11:10:24.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.411+0000 7f26293f3640 10 monclient: build_initial_monmap 2024-03-29T11:10:24.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.411+0000 7f26293f3640 10 monclient: monmap: 2024-03-29T11:10:24.391 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:10:24.391 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:10:24.391 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:10:24.391 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:10:24.391 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:24.391 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:24.391 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:10:24.392 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:10:24.392 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:10:24.392 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:24.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.412+0000 7f26293f3640 1 librados: starting msgr at 2024-03-29T11:10:24.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.412+0000 7f26293f3640 1 librados: starting objecter 2024-03-29T11:10:24.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.412+0000 7f26293f3640 1 Processor -- start 2024-03-29T11:10:24.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.413+0000 7f26293f3640 1 -- start start 2024-03-29T11:10:24.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.413+0000 7f26293f3640 1 librados: setting wanted keys 2024-03-29T11:10:24.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.413+0000 7f26293f3640 1 librados: calling monclient init 2024-03-29T11:10:24.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.413+0000 7f26293f3640 10 monclient: init 2024-03-29T11:10:24.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.414+0000 7f26293f3640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:24.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.415+0000 7f26293f3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:10:24.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.415+0000 7f26293f3640 10 monclient(hunting): picked mon.noname-a con 0x7f2624072f10 addr v1:172.21.15.71:6789/0 2024-03-29T11:10:24.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.415+0000 7f26293f3640 10 monclient(hunting): picked mon.noname-b con 0x7f26240acfa0 addr v1:172.21.15.137:6789/0 2024-03-29T11:10:24.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.415+0000 7f26293f3640 10 monclient(hunting): picked mon.noname-c con 0x7f26240ad380 addr v1:172.21.15.71:6790/0 2024-03-29T11:10:24.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.415+0000 7f26293f3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2624136b50 con 0x7f2624072f10 2024-03-29T11:10:24.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.415+0000 7f26293f3640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f262406f960 con 0x7f26240acfa0 2024-03-29T11:10:24.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.415+0000 7f26293f3640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f262406e810 con 0x7f26240ad380 2024-03-29T11:10:24.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.415+0000 7f26293f3640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:24.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.415+0000 7f26293f3640 10 monclient(hunting): authenticate will time out at 1380.673667s 2024-03-29T11:10:24.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.415+0000 7f2628bf2640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f26240ad380 0x7f2624136440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:47754/0 (socket says 172.21.15.71:47754) 2024-03-29T11:10:24.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.415+0000 7f2628bf2640 1 -- 172.21.15.71:0/1996035860 learned_addr learned my addr 172.21.15.71:0/1996035860 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:10:24.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.416+0000 7f2620ff9640 10 client.?.objecter ms_handle_connect 0x7f26240ad380 2024-03-29T11:10:24.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.416+0000 7f2620ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:24.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.418+0000 7f2620ff9640 1 -- 172.21.15.71:0/1996035860 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2309492507 0 0) 0x7f262406e810 con 0x7f26240ad380 2024-03-29T11:10:24.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.418+0000 7f2620ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:10:24.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.418+0000 7f2620ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:24.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.418+0000 7f2620ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:24.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.418+0000 7f2620ff9640 10 monclient(hunting): my global_id is 4655 2024-03-29T11:10:24.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.418+0000 7f2620ff9640 1 -- 172.21.15.71:0/1996035860 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2610004eb0 con 0x7f26240ad380 2024-03-29T11:10:24.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.419+0000 7f2620ff9640 10 client.?.objecter ms_handle_connect 0x7f2624072f10 2024-03-29T11:10:24.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.419+0000 7f2620ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:24.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.419+0000 7f2620ff9640 1 -- 172.21.15.71:0/1996035860 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340340667 0 0) 0x7f2624136b50 con 0x7f2624072f10 2024-03-29T11:10:24.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.419+0000 7f2620ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:10:24.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.419+0000 7f2620ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:24.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.419+0000 7f2620ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:24.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.419+0000 7f2620ff9640 10 monclient(hunting): my global_id is 4596 2024-03-29T11:10:24.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.419+0000 7f2620ff9640 1 -- 172.21.15.71:0/1996035860 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f262406e810 con 0x7f2624072f10 2024-03-29T11:10:24.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.419+0000 7f2620ff9640 10 client.?.objecter ms_handle_connect 0x7f26240acfa0 2024-03-29T11:10:24.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.419+0000 7f2620ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:24.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.419+0000 7f2620ff9640 1 -- 172.21.15.71:0/1996035860 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2721908856 0 0) 0x7f262406e810 con 0x7f2624072f10 2024-03-29T11:10:24.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 1 -- 172.21.15.71:0/1996035860 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2624136b50 con 0x7f2624072f10 2024-03-29T11:10:24.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 1 -- 172.21.15.71:0/1996035860 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2614004cf0 con 0x7f2624072f10 2024-03-29T11:10:24.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:24.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:10:24.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 10 monclient(hunting): dump: 2024-03-29T11:10:24.404 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:24.404 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:24.404 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:24.404 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:24.404 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:24.405 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:24.405 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:24.405 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:24.405 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:24.405 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:24.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:10:24.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 1 -- 172.21.15.71:0/1996035860 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216803626 0 0) 0x7f262406f960 con 0x7f26240acfa0 2024-03-29T11:10:24.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:10:24.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:24.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:24.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:10:24.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 1 -- 172.21.15.71:0/1996035860 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f262406e810 con 0x7f26240acfa0 2024-03-29T11:10:24.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 1 -- 172.21.15.71:0/1996035860 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2332805255 0 0) 0x7f2624136b50 con 0x7f2624072f10 2024-03-29T11:10:24.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 1 -- 172.21.15.71:0/1996035860 >> v1:172.21.15.71:6790/0 conn(0x7f26240ad380 legacy=0x7f2624136440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:24.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 1 -- 172.21.15.71:0/1996035860 >> v1:172.21.15.137:6789/0 conn(0x7f26240acfa0 legacy=0x7f2624135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:24.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:10:24.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 1 monclient: found mon.a 2024-03-29T11:10:24.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:24.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:10:24.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 1 -- 172.21.15.71:0/1996035860 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f262406e9a0 con 0x7f2624072f10 2024-03-29T11:10:24.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 10 monclient: _finish_auth 0 2024-03-29T11:10:24.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:10:24.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:24.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f26293f3640 5 monclient: authenticate success, global_id 4596 2024-03-29T11:10:24.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f26293f3640 10 monclient: _renew_subs 2024-03-29T11:10:24.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f26293f3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:10:24.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f26293f3640 1 -- 172.21.15.71:0/1996035860 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f262413d960 con 0x7f2624072f10 2024-03-29T11:10:24.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 1 -- 172.21.15.71:0/1996035860 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2614001900 con 0x7f2624072f10 2024-03-29T11:10:24.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:24.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 1 -- 172.21.15.71:0/1996035860 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f2614001c10 con 0x7f2624072f10 2024-03-29T11:10:24.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:24.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:10:24.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2620ff9640 10 monclient: dump: 2024-03-29T11:10:24.407 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:24.407 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:24.407 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:24.407 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:24.408 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:24.408 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:24.408 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:24.408 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:24.408 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:24.408 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:24.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f2621ffb640 4 set_mon_vals no callback set 2024-03-29T11:10:24.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f26293f3640 10 client.4596.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:10:24.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f26293f3640 10 monclient: _renew_subs 2024-03-29T11:10:24.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f26293f3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:10:24.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f26293f3640 1 -- 172.21.15.71:0/1996035860 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f262406f220 con 0x7f2624072f10 2024-03-29T11:10:24.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.420+0000 7f26293f3640 1 librados: init done 2024-03-29T11:10:24.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.421+0000 7f26293f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:24.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.422+0000 7f2620ff9640 1 -- 172.21.15.71:0/1996035860 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f261401aeb0 con 0x7f2624072f10 2024-03-29T11:10:24.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.422+0000 7f26293f3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:24.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.422+0000 7f26293f3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:10:24.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.422+0000 7f26293f3640 1 -- 172.21.15.71:0/1996035860 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f262404f730 con 0x7f2624072f10 2024-03-29T11:10:24.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.423+0000 7f2620ff9640 1 -- 172.21.15.71:0/1996035860 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f2614054920 con 0x7f2624072f10 2024-03-29T11:10:24.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.423+0000 7f2620ff9640 10 client.4596.objecter ms_dispatch 0x7f2624000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:10:24.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.423+0000 7f2620ff9640 3 client.4596.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:10:24.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.423+0000 7f2620ff9640 3 client.4596.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:10:24.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.423+0000 7f2620ff9640 20 client.4596.objecter dump_active .. 0 homeless 2024-03-29T11:10:24.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.423+0000 7f26237fe640 10 monclient: get_auth_request con 0x7f261006de20 auth_method 0 2024-03-29T11:10:24.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.425+0000 7f2620ff9640 10 client.4596.objecter ms_handle_connect 0x7f261006de20 2024-03-29T11:10:24.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.432+0000 7f2620ff9640 1 -- 172.21.15.71:0/1996035860 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2614050bc0 con 0x7f2624072f10 2024-03-29T11:10:24.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.432+0000 7f2620ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:24.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.432+0000 7f2620ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:10:24.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.756+0000 7f26293f3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:24.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.756+0000 7f26293f3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:24.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.756+0000 7f26293f3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:10:24.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.756+0000 7f26293f3640 1 -- 172.21.15.71:0/1996035860 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f262406eb30 con 0x7f2624072f10 2024-03-29T11:10:24.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.756+0000 7f2620ff9640 1 -- 172.21.15.71:0/1996035860 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f262406eb30 con 0x7f2624072f10 2024-03-29T11:10:24.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.757+0000 7f2620ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:24.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.757+0000 7f2620ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:10:24.735 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:10:24.735 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:10:24.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.760+0000 7f26293f3640 10 librados: watch_flush enter 2024-03-29T11:10:24.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.760+0000 7f26293f3640 10 librados: watch_flush exit 2024-03-29T11:10:24.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.760+0000 7f26293f3640 20 client.4596.objecter shutdown clearing up homeless session... 2024-03-29T11:10:24.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.760+0000 7f26293f3640 10 client.4596.objecter successfully canceled tick 2024-03-29T11:10:24.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.760+0000 7f26293f3640 1 -- 172.21.15.71:0/1996035860 >> v1:172.21.15.137:6816/973332665 conn(0x7f261006de20 legacy=0x7f26100702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:24.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.760+0000 7f26293f3640 10 monclient: shutdown 2024-03-29T11:10:24.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.760+0000 7f26293f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:24.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.760+0000 7f26293f3640 1 -- 172.21.15.71:0/1996035860 >> v1:172.21.15.71:6789/0 conn(0x7f2624072f10 legacy=0x7f26241db2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:24.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.760+0000 7f26293f3640 1 -- 172.21.15.71:0/1996035860 shutdown_connections 2024-03-29T11:10:24.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.760+0000 7f26293f3640 1 -- 172.21.15.71:0/1996035860 >> 172.21.15.71:0/1996035860 conn(0x7f26240b3a30 msgr2=0x7f26240b0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:24.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.761+0000 7f26293f3640 1 -- 172.21.15.71:0/1996035860 shutdown_connections 2024-03-29T11:10:24.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.761+0000 7f26293f3640 1 -- 172.21.15.71:0/1996035860 wait complete. 2024-03-29T11:10:24.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:24.761+0000 7f26293f3640 1 librados: shutdown 2024-03-29T11:10:24.755 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:10:31.562 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:10:31.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.732+0000 7f5e7af79640 10 monclient: get_monmap_and_config 2024-03-29T11:10:31.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.732+0000 7f5e7af79640 10 monclient: build_initial_monmap 2024-03-29T11:10:31.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.733+0000 7f5e7af79640 10 monclient: monmap: 2024-03-29T11:10:31.712 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:10:31.712 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:31.712 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:10:31.733836+0000 2024-03-29T11:10:31.712 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:10:31.733836+0000 2024-03-29T11:10:31.713 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:31.713 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:31.713 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:10:31.713 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:10:31.713 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:10:31.713 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:31.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.734+0000 7f5e7af79640 1 Processor -- start 2024-03-29T11:10:31.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.734+0000 7f5e7af79640 1 -- start start 2024-03-29T11:10:31.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.734+0000 7f5e7af79640 10 monclient: init 2024-03-29T11:10:31.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.735+0000 7f5e7af79640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:31.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.735+0000 7f5e7af79640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:10:31.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.735+0000 7f5e7af79640 10 monclient(hunting): picked mon.noname-b con 0x7f5e740b0860 addr v1:172.21.15.71:6790/0 2024-03-29T11:10:31.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.735+0000 7f5e7af79640 10 monclient(hunting): picked mon.noname-c con 0x7f5e7413ef30 addr v1:172.21.15.137:6789/0 2024-03-29T11:10:31.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.735+0000 7f5e7af79640 10 monclient(hunting): picked mon.noname-a con 0x7f5e7406d9d0 addr v1:172.21.15.71:6789/0 2024-03-29T11:10:31.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.735+0000 7f5e7af79640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e740b0c40 con 0x7f5e7406d9d0 2024-03-29T11:10:31.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.735+0000 7f5e7af79640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e740aed10 con 0x7f5e7413ef30 2024-03-29T11:10:31.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.735+0000 7f5e7af79640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e740ab1b0 con 0x7f5e740b0860 2024-03-29T11:10:31.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.735+0000 7f5e7af79640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:31.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.735+0000 7f5e7af79640 10 monclient(hunting): authenticate will time out at 1387.993310s 2024-03-29T11:10:31.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.735+0000 7f5e73fff640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f5e7413ef30 0x7f5e740af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:39012/0 (socket says 172.21.15.71:39012) 2024-03-29T11:10:31.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.735+0000 7f5e73fff640 1 -- 172.21.15.71:0/463167983 learned_addr learned my addr 172.21.15.71:0/463167983 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:10:31.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.736+0000 7f5e737fe640 1 -- 172.21.15.71:0/463167983 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 858758028 0 0) 0x7f5e740aed10 con 0x7f5e7413ef30 2024-03-29T11:10:31.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.736+0000 7f5e737fe640 10 monclient(hunting): _negotiate 2024-03-29T11:10:31.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.736+0000 7f5e737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:31.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.736+0000 7f5e737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:31.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.736+0000 7f5e737fe640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:10:31.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.736+0000 7f5e737fe640 1 -- 172.21.15.71:0/463167983 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e5c003520 con 0x7f5e7413ef30 2024-03-29T11:10:31.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.736+0000 7f5e737fe640 1 -- 172.21.15.71:0/463167983 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 354263936 0 0) 0x7f5e5c003520 con 0x7f5e7413ef30 2024-03-29T11:10:31.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.736+0000 7f5e737fe640 1 -- 172.21.15.71:0/463167983 >> v1:172.21.15.71:6790/0 conn(0x7f5e740b0860 legacy=0x7f5e7413e820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:10:31.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.738+0000 7f5e737fe640 1 -- 172.21.15.71:0/463167983 >> v1:172.21.15.71:6789/0 conn(0x7f5e7406d9d0 legacy=0x7f5e7414a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:10:31.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.738+0000 7f5e737fe640 10 monclient: _finish_hunting 0 2024-03-29T11:10:31.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.738+0000 7f5e737fe640 1 monclient: found mon.noname-c 2024-03-29T11:10:31.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.738+0000 7f5e737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:31.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.738+0000 7f5e737fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.137:6789/0 2024-03-29T11:10:31.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.738+0000 7f5e737fe640 1 -- 172.21.15.71:0/463167983 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e740ab490 con 0x7f5e7413ef30 2024-03-29T11:10:31.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.738+0000 7f5e737fe640 10 monclient: _finish_auth 0 2024-03-29T11:10:31.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.738+0000 7f5e737fe640 10 monclient: _check_auth_tickets 2024-03-29T11:10:31.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.738+0000 7f5e737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:31.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.738+0000 7f5e7af79640 5 monclient: authenticate success, global_id 4522 2024-03-29T11:10:31.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.738+0000 7f5e7af79640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:31.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.738+0000 7f5e737fe640 1 -- 172.21.15.71:0/463167983 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f5e6c001a10 con 0x7f5e7413ef30 2024-03-29T11:10:31.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.738+0000 7f5e737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:31.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.738+0000 7f5e737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:10:31.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.738+0000 7f5e737fe640 10 monclient: dump: 2024-03-29T11:10:31.718 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:31.718 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:31.718 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:31.718 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:31.718 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:31.718 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:31.719 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:31.719 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:31.719 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:31.719 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:31.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.738+0000 7f5e7af79640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:31.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.739+0000 7f5e737fe640 1 -- 172.21.15.71:0/463167983 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e6c001c70 con 0x7f5e7413ef30 2024-03-29T11:10:31.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.739+0000 7f5e737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:31.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.739+0000 7f5e7af79640 10 monclient: get_monmap_and_config success 2024-03-29T11:10:31.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.739+0000 7f5e7af79640 4 set_mon_vals no callback set 2024-03-29T11:10:31.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.740+0000 7f5e7af79640 10 monclient: shutdown 2024-03-29T11:10:31.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.740+0000 7f5e7af79640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:31.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.740+0000 7f5e7af79640 1 -- 172.21.15.71:0/463167983 >> v1:172.21.15.137:6789/0 conn(0x7f5e7413ef30 legacy=0x7f5e740af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:31.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.740+0000 7f5e737fe640 1 -- 172.21.15.71:0/463167983 <== mon.1 v1:172.21.15.137:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f5e6c002080 con 0x7f5e7413ef30 2024-03-29T11:10:31.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.740+0000 7f5e7af79640 1 -- 172.21.15.71:0/463167983 shutdown_connections 2024-03-29T11:10:31.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.740+0000 7f5e737fe640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:10:31.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.740+0000 7f5e7af79640 1 -- 172.21.15.71:0/463167983 >> 172.21.15.71:0/463167983 conn(0x7f5e7406b660 msgr2=0x7f5e7406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:31.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.742+0000 7f5e7af79640 1 -- 172.21.15.71:0/463167983 shutdown_connections 2024-03-29T11:10:31.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.744+0000 7f5e7af79640 1 -- 172.21.15.71:0/463167983 wait complete. 2024-03-29T11:10:31.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.746+0000 7f5e7af79640 10 monclient: build_initial_monmap 2024-03-29T11:10:31.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.746+0000 7f5e7af79640 10 monclient: monmap: 2024-03-29T11:10:31.725 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:10:31.725 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:10:31.725 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:10:31.725 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:10:31.725 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:31.725 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:31.725 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:10:31.725 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:10:31.725 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:10:31.725 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:31.726 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.746+0000 7f5e7af79640 1 librados: starting msgr at 2024-03-29T11:10:31.726 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.746+0000 7f5e7af79640 1 librados: starting objecter 2024-03-29T11:10:31.726 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.746+0000 7f5e7af79640 1 Processor -- start 2024-03-29T11:10:31.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.747+0000 7f5e7af79640 1 -- start start 2024-03-29T11:10:31.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.747+0000 7f5e7af79640 1 librados: setting wanted keys 2024-03-29T11:10:31.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.747+0000 7f5e7af79640 1 librados: calling monclient init 2024-03-29T11:10:31.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.747+0000 7f5e7af79640 10 monclient: init 2024-03-29T11:10:31.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.748+0000 7f5e7af79640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:31.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.748+0000 7f5e7af79640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:10:31.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.748+0000 7f5e7af79640 10 monclient(hunting): picked mon.noname-b con 0x7f5e7406d9d0 addr v1:172.21.15.137:6789/0 2024-03-29T11:10:31.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.748+0000 7f5e7af79640 10 monclient(hunting): picked mon.noname-a con 0x7f5e740b0860 addr v1:172.21.15.71:6789/0 2024-03-29T11:10:31.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.748+0000 7f5e7af79640 10 monclient(hunting): picked mon.noname-c con 0x7f5e741db2f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:10:31.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.748+0000 7f5e7af79640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e7406ed10 con 0x7f5e740b0860 2024-03-29T11:10:31.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.748+0000 7f5e7af79640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e7406e4a0 con 0x7f5e7406d9d0 2024-03-29T11:10:31.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.748+0000 7f5e7af79640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e74134d00 con 0x7f5e741db2f0 2024-03-29T11:10:31.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.748+0000 7f5e7af79640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:31.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.748+0000 7f5e7af79640 10 monclient(hunting): authenticate will time out at 1388.006305s 2024-03-29T11:10:31.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.748+0000 7f5e78d15640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f5e7406d9d0 0x7f5e741db970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:42248/0 (socket says 172.21.15.71:42248) 2024-03-29T11:10:31.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.749+0000 7f5e78d15640 1 -- 172.21.15.71:0/3251135690 learned_addr learned my addr 172.21.15.71:0/3251135690 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:10:31.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.750+0000 7f5e717fa640 10 client.?.objecter ms_handle_connect 0x7f5e7406d9d0 2024-03-29T11:10:31.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.750+0000 7f5e717fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:31.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.750+0000 7f5e717fa640 1 -- 172.21.15.71:0/3251135690 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1969581032 0 0) 0x7f5e7406e4a0 con 0x7f5e7406d9d0 2024-03-29T11:10:31.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.751+0000 7f5e717fa640 10 monclient(hunting): _negotiate 2024-03-29T11:10:31.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.751+0000 7f5e717fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:31.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.751+0000 7f5e717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:31.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.751+0000 7f5e717fa640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:10:31.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.751+0000 7f5e717fa640 1 -- 172.21.15.71:0/3251135690 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e60004eb0 con 0x7f5e7406d9d0 2024-03-29T11:10:31.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.752+0000 7f5e717fa640 10 client.?.objecter ms_handle_connect 0x7f5e741db2f0 2024-03-29T11:10:31.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.752+0000 7f5e717fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:31.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.752+0000 7f5e717fa640 1 -- 172.21.15.71:0/3251135690 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2365590708 0 0) 0x7f5e60004eb0 con 0x7f5e7406d9d0 2024-03-29T11:10:31.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.752+0000 7f5e717fa640 1 -- 172.21.15.71:0/3251135690 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e7406e4a0 con 0x7f5e7406d9d0 2024-03-29T11:10:31.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.752+0000 7f5e717fa640 1 -- 172.21.15.71:0/3251135690 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866379127 0 0) 0x7f5e74134d00 con 0x7f5e741db2f0 2024-03-29T11:10:31.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.752+0000 7f5e717fa640 10 monclient(hunting): _negotiate 2024-03-29T11:10:31.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.752+0000 7f5e717fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:31.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.752+0000 7f5e717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:31.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.752+0000 7f5e717fa640 10 monclient(hunting): my global_id is 4661 2024-03-29T11:10:31.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.752+0000 7f5e717fa640 1 -- 172.21.15.71:0/3251135690 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e60004eb0 con 0x7f5e741db2f0 2024-03-29T11:10:31.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.752+0000 7f5e717fa640 1 -- 172.21.15.71:0/3251135690 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f5e64001930 con 0x7f5e7406d9d0 2024-03-29T11:10:31.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.752+0000 7f5e717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:31.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.752+0000 7f5e717fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:10:31.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.752+0000 7f5e717fa640 10 monclient(hunting): dump: 2024-03-29T11:10:31.736 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:31.736 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:31.736 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:31.736 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:31.736 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:31.736 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:31.736 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:31.737 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:31.737 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:31.737 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:31.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.752+0000 7f5e717fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:10:31.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.753+0000 7f5e717fa640 1 -- 172.21.15.71:0/3251135690 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2596587952 0 0) 0x7f5e7406e4a0 con 0x7f5e7406d9d0 2024-03-29T11:10:31.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.753+0000 7f5e717fa640 1 -- 172.21.15.71:0/3251135690 >> v1:172.21.15.71:6790/0 conn(0x7f5e741db2f0 legacy=0x7f5e7406f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:31.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.753+0000 7f5e717fa640 1 -- 172.21.15.71:0/3251135690 >> v1:172.21.15.71:6789/0 conn(0x7f5e740b0860 legacy=0x7f5e74071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:31.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.753+0000 7f5e717fa640 10 monclient: _finish_hunting 0 2024-03-29T11:10:31.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.753+0000 7f5e717fa640 1 monclient: found mon.b 2024-03-29T11:10:31.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.753+0000 7f5e717fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:31.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.753+0000 7f5e717fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:10:31.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.753+0000 7f5e717fa640 1 -- 172.21.15.71:0/3251135690 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e741dc080 con 0x7f5e7406d9d0 2024-03-29T11:10:31.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.753+0000 7f5e717fa640 10 monclient: _finish_auth 0 2024-03-29T11:10:31.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.753+0000 7f5e717fa640 10 monclient: _check_auth_tickets 2024-03-29T11:10:31.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.753+0000 7f5e717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:31.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.753+0000 7f5e7af79640 5 monclient: authenticate success, global_id 4525 2024-03-29T11:10:31.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.753+0000 7f5e7af79640 10 monclient: _renew_subs 2024-03-29T11:10:31.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.753+0000 7f5e7af79640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:10:31.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.753+0000 7f5e7af79640 1 -- 172.21.15.71:0/3251135690 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e741dc270 con 0x7f5e7406d9d0 2024-03-29T11:10:31.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.753+0000 7f5e7af79640 10 client.4525.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:10:31.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.753+0000 7f5e7af79640 10 monclient: _renew_subs 2024-03-29T11:10:31.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.753+0000 7f5e7af79640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:10:31.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.753+0000 7f5e7af79640 1 -- 172.21.15.71:0/3251135690 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e74071b80 con 0x7f5e7406d9d0 2024-03-29T11:10:31.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.753+0000 7f5e7af79640 1 librados: init done 2024-03-29T11:10:31.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.755+0000 7f5e7af79640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:31.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.755+0000 7f5e717fa640 1 -- 172.21.15.71:0/3251135690 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e640035f0 con 0x7f5e7406d9d0 2024-03-29T11:10:31.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.755+0000 7f5e717fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:31.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.756+0000 7f5e717fa640 1 -- 172.21.15.71:0/3251135690 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f5e64003a00 con 0x7f5e7406d9d0 2024-03-29T11:10:31.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.756+0000 7f5e717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:31.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.756+0000 7f5e717fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:10:31.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.756+0000 7f5e717fa640 10 monclient: dump: 2024-03-29T11:10:31.739 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:31.739 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:31.739 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:31.739 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:31.739 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:31.740 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:31.740 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:31.740 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:31.740 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:31.740 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:31.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.756+0000 7f5e72ffd640 4 set_mon_vals no callback set 2024-03-29T11:10:31.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.757+0000 7f5e7af79640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:31.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.757+0000 7f5e7af79640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:10:31.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.757+0000 7f5e7af79640 1 -- 172.21.15.71:0/3251135690 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e7404eee0 con 0x7f5e7406d9d0 2024-03-29T11:10:31.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.757+0000 7f5e717fa640 1 -- 172.21.15.71:0/3251135690 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f5e64003c20 con 0x7f5e7406d9d0 2024-03-29T11:10:31.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.758+0000 7f5e717fa640 1 -- 172.21.15.71:0/3251135690 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2894269845 0 0) 0x7f5e64084980 con 0x7f5e7406d9d0 2024-03-29T11:10:31.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.758+0000 7f5e717fa640 10 client.4525.objecter ms_dispatch 0x7f5e74000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:10:31.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.758+0000 7f5e717fa640 3 client.4525.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:10:31.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.758+0000 7f5e717fa640 3 client.4525.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:10:31.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.758+0000 7f5e717fa640 20 client.4525.objecter dump_active .. 0 homeless 2024-03-29T11:10:31.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.758+0000 7f5e73fff640 10 monclient: get_auth_request con 0x7f5e6006dd70 auth_method 0 2024-03-29T11:10:31.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.761+0000 7f5e717fa640 10 client.4525.objecter ms_handle_connect 0x7f5e6006dd70 2024-03-29T11:10:31.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.767+0000 7f5e717fa640 1 -- 172.21.15.71:0/3251135690 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5e64050260 con 0x7f5e7406d9d0 2024-03-29T11:10:31.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.767+0000 7f5e717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:31.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:31.767+0000 7f5e717fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:10:32.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:32.096+0000 7f5e7af79640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:32.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:32.097+0000 7f5e7af79640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:32.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:32.097+0000 7f5e7af79640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:10:32.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:32.097+0000 7f5e7af79640 1 -- 172.21.15.71:0/3251135690 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e74075f10 con 0x7f5e7406d9d0 2024-03-29T11:10:32.077 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:10:32.077 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:00:34.381969+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:00:34.381969+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:10:32.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:32.097+0000 7f5e717fa640 1 -- 172.21.15.71:0/3251135690 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1891485329) 0x7f5e64052190 con 0x7f5e7406d9d0 2024-03-29T11:10:32.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:32.098+0000 7f5e717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:32.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:32.098+0000 7f5e717fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:10:32.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:32.101+0000 7f5e7af79640 10 librados: watch_flush enter 2024-03-29T11:10:32.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:32.101+0000 7f5e7af79640 10 librados: watch_flush exit 2024-03-29T11:10:32.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:32.101+0000 7f5e7af79640 20 client.4525.objecter shutdown clearing up homeless session... 2024-03-29T11:10:32.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:32.101+0000 7f5e7af79640 10 client.4525.objecter successfully canceled tick 2024-03-29T11:10:32.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:32.101+0000 7f5e7af79640 1 -- 172.21.15.71:0/3251135690 >> v1:172.21.15.137:6816/973332665 conn(0x7f5e6006dd70 legacy=0x7f5e60070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:32.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:32.101+0000 7f5e7af79640 10 monclient: shutdown 2024-03-29T11:10:32.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:32.101+0000 7f5e7af79640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:32.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:32.101+0000 7f5e7af79640 1 -- 172.21.15.71:0/3251135690 >> v1:172.21.15.137:6789/0 conn(0x7f5e7406d9d0 legacy=0x7f5e741db970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:32.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:32.102+0000 7f5e7af79640 1 -- 172.21.15.71:0/3251135690 shutdown_connections 2024-03-29T11:10:32.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:32.102+0000 7f5e7af79640 1 -- 172.21.15.71:0/3251135690 >> 172.21.15.71:0/3251135690 conn(0x7f5e7406b660 msgr2=0x7f5e7413d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:32.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:32.102+0000 7f5e7af79640 1 -- 172.21.15.71:0/3251135690 shutdown_connections 2024-03-29T11:10:32.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:32.102+0000 7f5e7af79640 1 -- 172.21.15.71:0/3251135690 wait complete. 2024-03-29T11:10:32.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:32.102+0000 7f5e7af79640 1 librados: shutdown 2024-03-29T11:10:32.097 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:10:38.904 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:10:39.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.073+0000 7f39f1b7b640 10 monclient: get_monmap_and_config 2024-03-29T11:10:39.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.073+0000 7f39f1b7b640 10 monclient: build_initial_monmap 2024-03-29T11:10:39.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.073+0000 7f39f1b7b640 10 monclient: monmap: 2024-03-29T11:10:39.052 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:10:39.052 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:39.052 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:10:39.074528+0000 2024-03-29T11:10:39.053 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:10:39.074528+0000 2024-03-29T11:10:39.053 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:39.053 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:39.053 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:10:39.053 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:10:39.053 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:10:39.053 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:39.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.075+0000 7f39f1b7b640 1 Processor -- start 2024-03-29T11:10:39.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.075+0000 7f39f1b7b640 1 -- start start 2024-03-29T11:10:39.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.075+0000 7f39f1b7b640 10 monclient: init 2024-03-29T11:10:39.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.076+0000 7f39f1b7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:39.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.076+0000 7f39f1b7b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:10:39.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.076+0000 7f39f1b7b640 10 monclient(hunting): picked mon.noname-c con 0x7f39ec0b0770 addr v1:172.21.15.137:6789/0 2024-03-29T11:10:39.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.076+0000 7f39f1b7b640 10 monclient(hunting): picked mon.noname-b con 0x7f39ec14a700 addr v1:172.21.15.71:6790/0 2024-03-29T11:10:39.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.076+0000 7f39f1b7b640 10 monclient(hunting): picked mon.noname-a con 0x7f39ec0ac5a0 addr v1:172.21.15.71:6789/0 2024-03-29T11:10:39.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.076+0000 7f39f1b7b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39ec0aba70 con 0x7f39ec0ac5a0 2024-03-29T11:10:39.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.076+0000 7f39f1b7b640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39ec0aecb0 con 0x7f39ec0b0770 2024-03-29T11:10:39.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.076+0000 7f39f1b7b640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39ec0af150 con 0x7f39ec14a700 2024-03-29T11:10:39.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.076+0000 7f39f1b7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:39.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.076+0000 7f39f1b7b640 10 monclient(hunting): authenticate will time out at 1395.334555s 2024-03-29T11:10:39.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.076+0000 7f39eaffd640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f39ec14a700 0x7f39ec0abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:55406/0 (socket says 172.21.15.71:55406) 2024-03-29T11:10:39.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.076+0000 7f39eaffd640 1 -- 172.21.15.71:0/3796471566 learned_addr learned my addr 172.21.15.71:0/3796471566 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:10:39.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.079+0000 7f39ea7fc640 1 -- 172.21.15.71:0/3796471566 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079681464 0 0) 0x7f39ec0af150 con 0x7f39ec14a700 2024-03-29T11:10:39.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.079+0000 7f39ea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:10:39.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.079+0000 7f39ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:39.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.079+0000 7f39ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:39.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.079+0000 7f39ea7fc640 10 monclient(hunting): my global_id is 4664 2024-03-29T11:10:39.057 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.079+0000 7f39ea7fc640 1 -- 172.21.15.71:0/3796471566 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39d4003520 con 0x7f39ec14a700 2024-03-29T11:10:39.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.080+0000 7f39ea7fc640 1 -- 172.21.15.71:0/3796471566 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2105425623 0 0) 0x7f39d4003520 con 0x7f39ec14a700 2024-03-29T11:10:39.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.080+0000 7f39ea7fc640 1 -- 172.21.15.71:0/3796471566 >> v1:172.21.15.137:6789/0 conn(0x7f39ec0b0770 legacy=0x7f39ec149ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:10:39.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.080+0000 7f39ea7fc640 1 -- 172.21.15.71:0/3796471566 >> v1:172.21.15.71:6789/0 conn(0x7f39ec0ac5a0 legacy=0x7f39ec151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:10:39.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.083+0000 7f39ea7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:10:39.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.083+0000 7f39ea7fc640 1 monclient: found mon.noname-b 2024-03-29T11:10:39.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.083+0000 7f39ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:39.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.083+0000 7f39ea7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:10:39.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.083+0000 7f39ea7fc640 1 -- 172.21.15.71:0/3796471566 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ec0ac9d0 con 0x7f39ec14a700 2024-03-29T11:10:39.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.083+0000 7f39ea7fc640 10 monclient: _finish_auth 0 2024-03-29T11:10:39.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.083+0000 7f39ea7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:10:39.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.083+0000 7f39ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:39.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.083+0000 7f39ea7fc640 1 -- 172.21.15.71:0/3796471566 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f39e4001a10 con 0x7f39ec14a700 2024-03-29T11:10:39.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.083+0000 7f39ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:39.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.083+0000 7f39ea7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:10:39.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.083+0000 7f39ea7fc640 10 monclient: dump: 2024-03-29T11:10:39.062 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:39.062 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:39.062 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:39.062 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:39.062 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:39.063 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:39.063 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:39.063 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:39.063 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:39.063 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:39.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.083+0000 7f39ea7fc640 1 -- 172.21.15.71:0/3796471566 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39e4001be0 con 0x7f39ec14a700 2024-03-29T11:10:39.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.084+0000 7f39f1b7b640 5 monclient: authenticate success, global_id 4664 2024-03-29T11:10:39.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.084+0000 7f39f1b7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:39.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.084+0000 7f39ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:39.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.084+0000 7f39f1b7b640 10 monclient: get_monmap_and_config success 2024-03-29T11:10:39.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.084+0000 7f39f1b7b640 4 set_mon_vals no callback set 2024-03-29T11:10:39.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.085+0000 7f39f1b7b640 10 monclient: shutdown 2024-03-29T11:10:39.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.085+0000 7f39f1b7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:39.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.085+0000 7f39f1b7b640 1 -- 172.21.15.71:0/3796471566 >> v1:172.21.15.71:6790/0 conn(0x7f39ec14a700 legacy=0x7f39ec0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:39.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.086+0000 7f39f1b7b640 1 -- 172.21.15.71:0/3796471566 shutdown_connections 2024-03-29T11:10:39.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.086+0000 7f39f1b7b640 1 -- 172.21.15.71:0/3796471566 >> 172.21.15.71:0/3796471566 conn(0x7f39ec06b660 msgr2=0x7f39ec06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:39.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.086+0000 7f39f1b7b640 1 -- 172.21.15.71:0/3796471566 shutdown_connections 2024-03-29T11:10:39.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.086+0000 7f39f1b7b640 1 -- 172.21.15.71:0/3796471566 wait complete. 2024-03-29T11:10:39.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.087+0000 7f39f1b7b640 10 monclient: build_initial_monmap 2024-03-29T11:10:39.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.087+0000 7f39f1b7b640 10 monclient: monmap: 2024-03-29T11:10:39.066 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:10:39.066 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:10:39.066 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:10:39.066 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:10:39.066 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:39.066 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:39.066 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:10:39.066 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:10:39.066 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:10:39.066 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:39.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.087+0000 7f39f1b7b640 1 librados: starting msgr at 2024-03-29T11:10:39.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.087+0000 7f39f1b7b640 1 librados: starting objecter 2024-03-29T11:10:39.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.088+0000 7f39f1b7b640 1 Processor -- start 2024-03-29T11:10:39.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.088+0000 7f39f1b7b640 1 -- start start 2024-03-29T11:10:39.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.088+0000 7f39f1b7b640 1 librados: setting wanted keys 2024-03-29T11:10:39.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.088+0000 7f39f1b7b640 1 librados: calling monclient init 2024-03-29T11:10:39.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.088+0000 7f39f1b7b640 10 monclient: init 2024-03-29T11:10:39.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.089+0000 7f39f1b7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:39.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.089+0000 7f39f1b7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:10:39.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.089+0000 7f39f1b7b640 10 monclient(hunting): picked mon.noname-b con 0x7f39ec0ac5a0 addr v1:172.21.15.137:6789/0 2024-03-29T11:10:39.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.089+0000 7f39f1b7b640 10 monclient(hunting): picked mon.noname-a con 0x7f39ec0b0770 addr v1:172.21.15.71:6789/0 2024-03-29T11:10:39.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.089+0000 7f39f1b7b640 10 monclient(hunting): picked mon.noname-c con 0x7f39ec14a700 addr v1:172.21.15.71:6790/0 2024-03-29T11:10:39.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.089+0000 7f39f1b7b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39ec1261f0 con 0x7f39ec0b0770 2024-03-29T11:10:39.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.089+0000 7f39f1b7b640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39ec145b20 con 0x7f39ec0ac5a0 2024-03-29T11:10:39.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.089+0000 7f39f1b7b640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39ec125d10 con 0x7f39ec14a700 2024-03-29T11:10:39.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.089+0000 7f39f1b7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:39.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.089+0000 7f39f1b7b640 10 monclient(hunting): authenticate will time out at 1395.347246s 2024-03-29T11:10:39.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.091+0000 7f39ebfff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f39ec14a700 0x7f39ec126e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:39692/0 (socket says 172.21.15.71:39692) 2024-03-29T11:10:39.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.091+0000 7f39ebfff640 1 -- 172.21.15.71:0/3360589410 learned_addr learned my addr 172.21.15.71:0/3360589410 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:10:39.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.092+0000 7f39d3fff640 10 client.?.objecter ms_handle_connect 0x7f39ec0b0770 2024-03-29T11:10:39.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.092+0000 7f39d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:39.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.092+0000 7f39d3fff640 10 client.?.objecter ms_handle_connect 0x7f39ec14a700 2024-03-29T11:10:39.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.092+0000 7f39d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:39.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.093+0000 7f39d3fff640 1 -- 172.21.15.71:0/3360589410 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 62712542 0 0) 0x7f39ec1261f0 con 0x7f39ec0b0770 2024-03-29T11:10:39.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.093+0000 7f39d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:10:39.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.093+0000 7f39d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:39.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.093+0000 7f39d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:39.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.093+0000 7f39d3fff640 10 monclient(hunting): my global_id is 4602 2024-03-29T11:10:39.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.093+0000 7f39d3fff640 1 -- 172.21.15.71:0/3360589410 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39d8004eb0 con 0x7f39ec0b0770 2024-03-29T11:10:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.095+0000 7f39d3fff640 10 client.?.objecter ms_handle_connect 0x7f39ec0ac5a0 2024-03-29T11:10:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.095+0000 7f39d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.095+0000 7f39d3fff640 1 -- 172.21.15.71:0/3360589410 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1224044973 0 0) 0x7f39ec145b20 con 0x7f39ec0ac5a0 2024-03-29T11:10:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.095+0000 7f39d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:10:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.095+0000 7f39d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.095+0000 7f39d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.095+0000 7f39d3fff640 10 monclient(hunting): my global_id is 4528 2024-03-29T11:10:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.095+0000 7f39d3fff640 1 -- 172.21.15.71:0/3360589410 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39ec1261f0 con 0x7f39ec0ac5a0 2024-03-29T11:10:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.095+0000 7f39d3fff640 1 -- 172.21.15.71:0/3360589410 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1569474382 0 0) 0x7f39d8004eb0 con 0x7f39ec0b0770 2024-03-29T11:10:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.095+0000 7f39d3fff640 1 -- 172.21.15.71:0/3360589410 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39ec145b20 con 0x7f39ec0b0770 2024-03-29T11:10:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.095+0000 7f39d3fff640 1 -- 172.21.15.71:0/3360589410 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398622814 0 0) 0x7f39ec125d10 con 0x7f39ec14a700 2024-03-29T11:10:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.095+0000 7f39d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:10:39.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.095+0000 7f39d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:39.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.095+0000 7f39d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:39.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.095+0000 7f39d3fff640 10 monclient(hunting): my global_id is 4667 2024-03-29T11:10:39.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.095+0000 7f39d3fff640 1 -- 172.21.15.71:0/3360589410 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39d8004eb0 con 0x7f39ec14a700 2024-03-29T11:10:39.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.095+0000 7f39d3fff640 1 -- 172.21.15.71:0/3360589410 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f39e4004d00 con 0x7f39ec0b0770 2024-03-29T11:10:39.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.095+0000 7f39d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:39.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.095+0000 7f39d3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:10:39.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.096+0000 7f39d3fff640 10 monclient(hunting): dump: 2024-03-29T11:10:39.076 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:39.076 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:39.076 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:39.076 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:39.076 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:39.076 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:39.077 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:39.077 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:39.077 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:39.077 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:39.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.096+0000 7f39d3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:10:39.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.096+0000 7f39d3fff640 1 -- 172.21.15.71:0/3360589410 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3538824503 0 0) 0x7f39ec145b20 con 0x7f39ec0b0770 2024-03-29T11:10:39.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.096+0000 7f39d3fff640 1 -- 172.21.15.71:0/3360589410 >> v1:172.21.15.71:6790/0 conn(0x7f39ec14a700 legacy=0x7f39ec126e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:39.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.096+0000 7f39d3fff640 1 -- 172.21.15.71:0/3360589410 >> v1:172.21.15.137:6789/0 conn(0x7f39ec0ac5a0 legacy=0x7f39ec144e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:39.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.096+0000 7f39d3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:10:39.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.096+0000 7f39d3fff640 1 monclient: found mon.a 2024-03-29T11:10:39.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.096+0000 7f39d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:39.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.096+0000 7f39d3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:10:39.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.096+0000 7f39d3fff640 1 -- 172.21.15.71:0/3360589410 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ec125ea0 con 0x7f39ec0b0770 2024-03-29T11:10:39.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.096+0000 7f39d3fff640 10 monclient: _finish_auth 0 2024-03-29T11:10:39.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.096+0000 7f39d3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:10:39.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.096+0000 7f39d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:39.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.096+0000 7f39f1b7b640 5 monclient: authenticate success, global_id 4602 2024-03-29T11:10:39.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.096+0000 7f39f1b7b640 10 monclient: _renew_subs 2024-03-29T11:10:39.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.096+0000 7f39f1b7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:10:39.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.096+0000 7f39f1b7b640 1 -- 172.21.15.71:0/3360589410 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39ec1dc0d0 con 0x7f39ec0b0770 2024-03-29T11:10:39.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.096+0000 7f39f1b7b640 10 client.4602.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:10:39.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.096+0000 7f39f1b7b640 10 monclient: _renew_subs 2024-03-29T11:10:39.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.096+0000 7f39f1b7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:10:39.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.096+0000 7f39f1b7b640 1 -- 172.21.15.71:0/3360589410 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39ec1db870 con 0x7f39ec0b0770 2024-03-29T11:10:39.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.096+0000 7f39f1b7b640 1 librados: init done 2024-03-29T11:10:39.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.097+0000 7f39d3fff640 1 -- 172.21.15.71:0/3360589410 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39e4001e90 con 0x7f39ec0b0770 2024-03-29T11:10:39.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.097+0000 7f39d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:39.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.097+0000 7f39e9ffb640 4 set_mon_vals no callback set 2024-03-29T11:10:39.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.097+0000 7f39f1b7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:39.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.097+0000 7f39d3fff640 1 -- 172.21.15.71:0/3360589410 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f39e40052b0 con 0x7f39ec0b0770 2024-03-29T11:10:39.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.098+0000 7f39f1b7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:39.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.098+0000 7f39f1b7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:10:39.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.098+0000 7f39f1b7b640 1 -- 172.21.15.71:0/3360589410 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39cc005080 con 0x7f39ec0b0770 2024-03-29T11:10:39.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.099+0000 7f39d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:39.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.099+0000 7f39d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:10:39.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.099+0000 7f39d3fff640 10 monclient: dump: 2024-03-29T11:10:39.081 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:39.081 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:39.081 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:39.081 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:39.081 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:39.081 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:39.082 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:39.082 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:39.082 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:39.082 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:39.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.099+0000 7f39d3fff640 1 -- 172.21.15.71:0/3360589410 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f39e4005410 con 0x7f39ec0b0770 2024-03-29T11:10:39.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.099+0000 7f39d3fff640 1 -- 172.21.15.71:0/3360589410 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 45029510 0 0) 0x7f39e4084ed0 con 0x7f39ec0b0770 2024-03-29T11:10:39.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.099+0000 7f39d3fff640 10 client.4602.objecter ms_dispatch 0x7f39ec000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:10:39.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.099+0000 7f39d3fff640 3 client.4602.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:10:39.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.099+0000 7f39d3fff640 3 client.4602.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:10:39.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.099+0000 7f39d3fff640 20 client.4602.objecter dump_active .. 0 homeless 2024-03-29T11:10:39.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.099+0000 7f39eb7fe640 10 monclient: get_auth_request con 0x7f39d806dea0 auth_method 0 2024-03-29T11:10:39.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.100+0000 7f39d3fff640 10 client.4602.objecter ms_handle_connect 0x7f39d806dea0 2024-03-29T11:10:39.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.107+0000 7f39d3fff640 1 -- 172.21.15.71:0/3360589410 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f39e40507b0 con 0x7f39ec0b0770 2024-03-29T11:10:39.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.107+0000 7f39d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:39.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.107+0000 7f39d3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:10:39.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.426+0000 7f39f1b7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:39.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.426+0000 7f39f1b7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:39.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.426+0000 7f39f1b7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:10:39.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.426+0000 7f39f1b7b640 1 -- 172.21.15.71:0/3360589410 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39cc005330 con 0x7f39ec0b0770 2024-03-29T11:10:39.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.427+0000 7f39d3fff640 1 -- 172.21.15.71:0/3360589410 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 577002532) 0x7f39e40526e0 con 0x7f39ec0b0770 2024-03-29T11:10:39.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.427+0000 7f39d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:39.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.427+0000 7f39d3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:10:39.406 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:10:39.406 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:10:37.063243+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:10:37.063243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:10:39.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.430+0000 7f39f1b7b640 10 librados: watch_flush enter 2024-03-29T11:10:39.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.430+0000 7f39f1b7b640 10 librados: watch_flush exit 2024-03-29T11:10:39.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.430+0000 7f39f1b7b640 20 client.4602.objecter shutdown clearing up homeless session... 2024-03-29T11:10:39.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.430+0000 7f39f1b7b640 10 client.4602.objecter successfully canceled tick 2024-03-29T11:10:39.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.431+0000 7f39f1b7b640 1 -- 172.21.15.71:0/3360589410 >> v1:172.21.15.137:6816/973332665 conn(0x7f39d806dea0 legacy=0x7f39d8070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:39.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.431+0000 7f39f1b7b640 10 monclient: shutdown 2024-03-29T11:10:39.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.431+0000 7f39f1b7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:39.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.431+0000 7f39f1b7b640 1 -- 172.21.15.71:0/3360589410 >> v1:172.21.15.71:6789/0 conn(0x7f39ec0b0770 legacy=0x7f39ec126710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:39.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.433+0000 7f39f1b7b640 1 -- 172.21.15.71:0/3360589410 shutdown_connections 2024-03-29T11:10:39.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.433+0000 7f39f1b7b640 1 -- 172.21.15.71:0/3360589410 >> 172.21.15.71:0/3360589410 conn(0x7f39ec06b660 msgr2=0x7f39ec0b2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:39.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.433+0000 7f39f1b7b640 1 -- 172.21.15.71:0/3360589410 shutdown_connections 2024-03-29T11:10:39.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.433+0000 7f39f1b7b640 1 -- 172.21.15.71:0/3360589410 wait complete. 2024-03-29T11:10:39.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:39.434+0000 7f39f1b7b640 1 librados: shutdown 2024-03-29T11:10:39.430 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:10:46.237 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:10:46.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.402+0000 7efea5099640 10 monclient: get_monmap_and_config 2024-03-29T11:10:46.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.402+0000 7efea5099640 10 monclient: build_initial_monmap 2024-03-29T11:10:46.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.402+0000 7efea5099640 10 monclient: monmap: 2024-03-29T11:10:46.381 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:10:46.381 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:46.381 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:10:46.403061+0000 2024-03-29T11:10:46.381 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:10:46.403061+0000 2024-03-29T11:10:46.381 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:46.381 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:46.381 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:10:46.381 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:10:46.381 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:10:46.382 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:46.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.403+0000 7efea5099640 1 Processor -- start 2024-03-29T11:10:46.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.403+0000 7efea5099640 1 -- start start 2024-03-29T11:10:46.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.403+0000 7efea5099640 10 monclient: init 2024-03-29T11:10:46.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.404+0000 7efea5099640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:46.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.404+0000 7efea5099640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:10:46.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.404+0000 7efea5099640 10 monclient(hunting): picked mon.noname-b con 0x7efe980066e0 addr v1:172.21.15.71:6790/0 2024-03-29T11:10:46.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.404+0000 7efe9ffff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7efe980066e0 0x7efe980cf110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:45890/0 (socket says 172.21.15.71:45890) 2024-03-29T11:10:46.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.405+0000 7efea5099640 10 monclient(hunting): picked mon.noname-c con 0x7efe980cf820 addr v1:172.21.15.137:6789/0 2024-03-29T11:10:46.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.405+0000 7efea5099640 10 monclient(hunting): picked mon.noname-a con 0x7efe980c92b0 addr v1:172.21.15.71:6789/0 2024-03-29T11:10:46.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.405+0000 7efea5099640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9802c100 con 0x7efe980c92b0 2024-03-29T11:10:46.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.405+0000 7efea5099640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9802c3e0 con 0x7efe980cf820 2024-03-29T11:10:46.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.405+0000 7efea5099640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9802a030 con 0x7efe980066e0 2024-03-29T11:10:46.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.405+0000 7efea5099640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:46.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.405+0000 7efea5099640 10 monclient(hunting): authenticate will time out at 1402.663197s 2024-03-29T11:10:46.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.405+0000 7efe9ffff640 1 -- 172.21.15.71:0/1432124568 learned_addr learned my addr 172.21.15.71:0/1432124568 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:10:46.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.407+0000 7efe9effd640 1 -- 172.21.15.71:0/1432124568 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916554807 0 0) 0x7efe9802a030 con 0x7efe980066e0 2024-03-29T11:10:46.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.407+0000 7efe9effd640 10 monclient(hunting): _negotiate 2024-03-29T11:10:46.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.407+0000 7efe9effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:46.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.407+0000 7efe9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:46.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.407+0000 7efe9effd640 10 monclient(hunting): my global_id is 4670 2024-03-29T11:10:46.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.407+0000 7efe9effd640 1 -- 172.21.15.71:0/1432124568 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe90003520 con 0x7efe980066e0 2024-03-29T11:10:46.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.408+0000 7efe9effd640 1 -- 172.21.15.71:0/1432124568 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 27153735 0 0) 0x7efe90003520 con 0x7efe980066e0 2024-03-29T11:10:46.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.408+0000 7efe9effd640 1 -- 172.21.15.71:0/1432124568 >> v1:172.21.15.137:6789/0 conn(0x7efe980cf820 legacy=0x7efe980c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:46.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.408+0000 7efe9effd640 1 -- 172.21.15.71:0/1432124568 >> v1:172.21.15.71:6789/0 conn(0x7efe980c92b0 legacy=0x7efe980d74b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:10:46.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.409+0000 7efe9effd640 10 monclient: _finish_hunting 0 2024-03-29T11:10:46.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.409+0000 7efe9effd640 1 monclient: found mon.noname-b 2024-03-29T11:10:46.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.409+0000 7efe9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:46.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.409+0000 7efe9effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:10:46.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.409+0000 7efe9effd640 1 -- 172.21.15.71:0/1432124568 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe98029ce0 con 0x7efe980066e0 2024-03-29T11:10:46.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.409+0000 7efe9effd640 10 monclient: _finish_auth 0 2024-03-29T11:10:46.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.409+0000 7efe9effd640 10 monclient: _check_auth_tickets 2024-03-29T11:10:46.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.409+0000 7efe9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:46.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.409+0000 7efe9effd640 1 -- 172.21.15.71:0/1432124568 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7efe8c001690 con 0x7efe980066e0 2024-03-29T11:10:46.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.409+0000 7efe9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:46.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.409+0000 7efe9effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:10:46.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.409+0000 7efe9effd640 10 monclient: dump: 2024-03-29T11:10:46.389 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:46.389 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:46.389 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:46.389 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:46.389 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:46.389 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:46.389 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:46.389 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:46.389 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:46.389 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:46.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.409+0000 7efea5099640 5 monclient: authenticate success, global_id 4670 2024-03-29T11:10:46.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.409+0000 7efea5099640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:46.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.412+0000 7efe9effd640 1 -- 172.21.15.71:0/1432124568 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe8c001860 con 0x7efe980066e0 2024-03-29T11:10:46.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.412+0000 7efe9effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:46.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.412+0000 7efe9effd640 1 -- 172.21.15.71:0/1432124568 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7efe8c001cb0 con 0x7efe980066e0 2024-03-29T11:10:46.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.412+0000 7efe9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:46.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.412+0000 7efe9effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:10:46.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.412+0000 7efe9effd640 10 monclient: dump: 2024-03-29T11:10:46.391 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:46.391 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:46.391 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:46.391 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:46.391 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:46.391 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:46.391 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:46.391 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:46.391 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:46.391 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:46.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.414+0000 7efea5099640 10 monclient: get_monmap_and_config success 2024-03-29T11:10:46.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.414+0000 7efea5099640 4 set_mon_vals no callback set 2024-03-29T11:10:46.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.415+0000 7efea5099640 10 monclient: shutdown 2024-03-29T11:10:46.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.415+0000 7efea5099640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:46.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.415+0000 7efea5099640 1 -- 172.21.15.71:0/1432124568 >> v1:172.21.15.71:6790/0 conn(0x7efe980066e0 legacy=0x7efe980cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:46.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.416+0000 7efea5099640 1 -- 172.21.15.71:0/1432124568 shutdown_connections 2024-03-29T11:10:46.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.416+0000 7efea5099640 1 -- 172.21.15.71:0/1432124568 >> 172.21.15.71:0/1432124568 conn(0x7efe980058f0 msgr2=0x7efe98005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:46.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.416+0000 7efea5099640 1 -- 172.21.15.71:0/1432124568 shutdown_connections 2024-03-29T11:10:46.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.416+0000 7efea5099640 1 -- 172.21.15.71:0/1432124568 wait complete. 2024-03-29T11:10:46.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.417+0000 7efea5099640 10 monclient: build_initial_monmap 2024-03-29T11:10:46.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.417+0000 7efea5099640 10 monclient: monmap: 2024-03-29T11:10:46.396 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:10:46.396 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:10:46.396 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:10:46.396 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:10:46.396 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:46.396 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:46.396 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:10:46.396 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:10:46.396 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:10:46.396 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:46.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.418+0000 7efea5099640 1 librados: starting msgr at 2024-03-29T11:10:46.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.418+0000 7efea5099640 1 librados: starting objecter 2024-03-29T11:10:46.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.418+0000 7efea5099640 1 Processor -- start 2024-03-29T11:10:46.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.419+0000 7efea5099640 1 -- start start 2024-03-29T11:10:46.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.419+0000 7efea5099640 1 librados: setting wanted keys 2024-03-29T11:10:46.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.419+0000 7efea5099640 1 librados: calling monclient init 2024-03-29T11:10:46.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.419+0000 7efea5099640 10 monclient: init 2024-03-29T11:10:46.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.419+0000 7efea5099640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:46.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.419+0000 7efea5099640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:10:46.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.421+0000 7efea5099640 10 monclient(hunting): picked mon.noname-b con 0x7efe980066e0 addr v1:172.21.15.137:6789/0 2024-03-29T11:10:46.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.421+0000 7efea5099640 10 monclient(hunting): picked mon.noname-a con 0x7efe980c92b0 addr v1:172.21.15.71:6789/0 2024-03-29T11:10:46.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.421+0000 7efea5099640 10 monclient(hunting): picked mon.noname-c con 0x7efe980cf820 addr v1:172.21.15.71:6790/0 2024-03-29T11:10:46.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.421+0000 7efea5099640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9802e040 con 0x7efe980c92b0 2024-03-29T11:10:46.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.421+0000 7efea5099640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9802f230 con 0x7efe980066e0 2024-03-29T11:10:46.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.421+0000 7efea5099640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9802d000 con 0x7efe980cf820 2024-03-29T11:10:46.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.421+0000 7efea5099640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:46.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.421+0000 7efea5099640 10 monclient(hunting): authenticate will time out at 1402.679233s 2024-03-29T11:10:46.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.423+0000 7efea4898640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7efe980cf820 0x7efe98161290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:37738/0 (socket says 172.21.15.71:37738) 2024-03-29T11:10:46.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.423+0000 7efea4898640 1 -- 172.21.15.71:0/2251826146 learned_addr learned my addr 172.21.15.71:0/2251826146 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:10:46.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.423+0000 7efe9cff9640 10 client.?.objecter ms_handle_connect 0x7efe980cf820 2024-03-29T11:10:46.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.423+0000 7efe9cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:46.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.425+0000 7efe9cff9640 10 client.?.objecter ms_handle_connect 0x7efe980c92b0 2024-03-29T11:10:46.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.425+0000 7efe9cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:46.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.425+0000 7efe9cff9640 10 client.?.objecter ms_handle_connect 0x7efe980066e0 2024-03-29T11:10:46.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.425+0000 7efe9cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:46.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.425+0000 7efe9cff9640 1 -- 172.21.15.71:0/2251826146 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 785976474 0 0) 0x7efe9802d000 con 0x7efe980cf820 2024-03-29T11:10:46.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.425+0000 7efe9cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:10:46.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.425+0000 7efe9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:46.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.425+0000 7efe9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:46.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.425+0000 7efe9cff9640 10 monclient(hunting): my global_id is 4673 2024-03-29T11:10:46.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.425+0000 7efe9cff9640 1 -- 172.21.15.71:0/2251826146 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe84004eb0 con 0x7efe980cf820 2024-03-29T11:10:46.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.425+0000 7efe9cff9640 1 -- 172.21.15.71:0/2251826146 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3584771640 0 0) 0x7efe9802e040 con 0x7efe980c92b0 2024-03-29T11:10:46.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.425+0000 7efe9cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:10:46.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.425+0000 7efe9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:46.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.425+0000 7efe9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:46.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.425+0000 7efe9cff9640 10 monclient(hunting): my global_id is 4605 2024-03-29T11:10:46.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.425+0000 7efe9cff9640 1 -- 172.21.15.71:0/2251826146 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe9802d000 con 0x7efe980c92b0 2024-03-29T11:10:46.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.425+0000 7efe9cff9640 1 -- 172.21.15.71:0/2251826146 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3215399488 0 0) 0x7efe9802f230 con 0x7efe980066e0 2024-03-29T11:10:46.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.425+0000 7efe9cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:10:46.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.425+0000 7efe9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:46.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.425+0000 7efe9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:46.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.425+0000 7efe9cff9640 10 monclient(hunting): my global_id is 4531 2024-03-29T11:10:46.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.425+0000 7efe9cff9640 1 -- 172.21.15.71:0/2251826146 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe9802e040 con 0x7efe980066e0 2024-03-29T11:10:46.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 1 -- 172.21.15.71:0/2251826146 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 520633415 0 0) 0x7efe84004eb0 con 0x7efe980cf820 2024-03-29T11:10:46.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 1 -- 172.21.15.71:0/2251826146 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe9802f230 con 0x7efe980cf820 2024-03-29T11:10:46.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 1 -- 172.21.15.71:0/2251826146 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7efea004fdd0 con 0x7efe980cf820 2024-03-29T11:10:46.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:46.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:10:46.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 10 monclient(hunting): dump: 2024-03-29T11:10:46.411 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:46.411 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:46.412 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:46.412 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:46.412 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:46.412 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:46.412 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:46.412 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:46.412 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:46.412 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:46.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:10:46.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 1 -- 172.21.15.71:0/2251826146 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1976090182 0 0) 0x7efe9802d000 con 0x7efe980c92b0 2024-03-29T11:10:46.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 1 -- 172.21.15.71:0/2251826146 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe84004eb0 con 0x7efe980c92b0 2024-03-29T11:10:46.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 1 -- 172.21.15.71:0/2251826146 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7efe94001810 con 0x7efe980c92b0 2024-03-29T11:10:46.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:46.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:10:46.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 10 monclient(hunting): dump: 2024-03-29T11:10:46.413 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:46.413 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:46.413 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:46.413 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:46.413 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:46.413 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:46.413 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:46.413 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:46.413 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:46.413 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:46.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 1 -- 172.21.15.71:0/2251826146 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1768297060 0 0) 0x7efe9802e040 con 0x7efe980066e0 2024-03-29T11:10:46.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 1 -- 172.21.15.71:0/2251826146 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe9802d000 con 0x7efe980066e0 2024-03-29T11:10:46.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 1 -- 172.21.15.71:0/2251826146 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7efe8c0019d0 con 0x7efe980066e0 2024-03-29T11:10:46.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:46.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:10:46.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 10 monclient(hunting): dump: 2024-03-29T11:10:46.414 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:46.414 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:46.414 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:46.414 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:46.414 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:46.414 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:46.414 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:46.415 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:46.415 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:46.415 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:46.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 1 -- 172.21.15.71:0/2251826146 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1205155129 0 0) 0x7efe84004eb0 con 0x7efe980c92b0 2024-03-29T11:10:46.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 1 -- 172.21.15.71:0/2251826146 >> v1:172.21.15.71:6790/0 conn(0x7efe980cf820 legacy=0x7efe98161290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:46.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 1 -- 172.21.15.71:0/2251826146 >> v1:172.21.15.137:6789/0 conn(0x7efe980066e0 legacy=0x7efe9802d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:46.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:10:46.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 1 monclient: found mon.a 2024-03-29T11:10:46.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:46.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:10:46.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 1 -- 172.21.15.71:0/2251826146 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9802d2e0 con 0x7efe980c92b0 2024-03-29T11:10:46.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 10 monclient: _finish_auth 0 2024-03-29T11:10:46.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:10:46.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efe9cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:46.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efea5099640 5 monclient: authenticate success, global_id 4605 2024-03-29T11:10:46.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.426+0000 7efea5099640 10 monclient: _renew_subs 2024-03-29T11:10:46.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.427+0000 7efea5099640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:10:46.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.427+0000 7efea5099640 1 -- 172.21.15.71:0/2251826146 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe9816c2f0 con 0x7efe980c92b0 2024-03-29T11:10:46.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.427+0000 7efe9cff9640 1 -- 172.21.15.71:0/2251826146 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe94001d00 con 0x7efe980c92b0 2024-03-29T11:10:46.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.427+0000 7efea5099640 10 client.4605.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:10:46.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.427+0000 7efea5099640 10 monclient: _renew_subs 2024-03-29T11:10:46.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.427+0000 7efea5099640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:10:46.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.427+0000 7efea5099640 1 -- 172.21.15.71:0/2251826146 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe9816c6c0 con 0x7efe980c92b0 2024-03-29T11:10:46.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.427+0000 7efea5099640 1 librados: init done 2024-03-29T11:10:46.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.427+0000 7efe9cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:46.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.427+0000 7efe9cff9640 1 -- 172.21.15.71:0/2251826146 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7efe94003790 con 0x7efe980c92b0 2024-03-29T11:10:46.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.427+0000 7efe9cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:46.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.427+0000 7efe9cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:10:46.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.427+0000 7efe9cff9640 10 monclient: dump: 2024-03-29T11:10:46.417 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:46.417 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:46.417 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:46.417 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:46.417 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:46.417 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:46.417 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:46.417 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:46.418 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:46.418 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:46.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.428+0000 7efe9e7fc640 4 set_mon_vals no callback set 2024-03-29T11:10:46.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.428+0000 7efea5099640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:46.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.429+0000 7efea5099640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:46.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.429+0000 7efea5099640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:10:46.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.429+0000 7efea5099640 1 -- 172.21.15.71:0/2251826146 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe80005080 con 0x7efe980c92b0 2024-03-29T11:10:46.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.429+0000 7efe9cff9640 1 -- 172.21.15.71:0/2251826146 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7efe94019760 con 0x7efe980c92b0 2024-03-29T11:10:46.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.430+0000 7efe9ffff640 10 monclient: get_auth_request con 0x7efe8406e080 auth_method 0 2024-03-29T11:10:46.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.430+0000 7efe9cff9640 1 -- 172.21.15.71:0/2251826146 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 45029510 0 0) 0x7efe940837e0 con 0x7efe980c92b0 2024-03-29T11:10:46.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.430+0000 7efe9cff9640 10 client.4605.objecter ms_dispatch 0x7efea0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:10:46.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.430+0000 7efe9cff9640 3 client.4605.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:10:46.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.430+0000 7efe9cff9640 3 client.4605.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:10:46.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.430+0000 7efe9cff9640 20 client.4605.objecter dump_active .. 0 homeless 2024-03-29T11:10:46.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.431+0000 7efe9cff9640 10 client.4605.objecter ms_handle_connect 0x7efe8406e080 2024-03-29T11:10:46.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.437+0000 7efe9cff9640 1 -- 172.21.15.71:0/2251826146 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe9404f0c0 con 0x7efe980c92b0 2024-03-29T11:10:46.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.437+0000 7efe9cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:46.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.437+0000 7efe9cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:10:46.743 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.765+0000 7efea5099640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:46.743 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.765+0000 7efea5099640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:46.743 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.765+0000 7efea5099640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:10:46.743 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.765+0000 7efea5099640 1 -- 172.21.15.71:0/2251826146 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe80005330 con 0x7efe980c92b0 2024-03-29T11:10:46.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.766+0000 7efe9cff9640 1 -- 172.21.15.71:0/2251826146 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 577002532) 0x7efe94050ff0 con 0x7efe980c92b0 2024-03-29T11:10:46.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.766+0000 7efe9cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:46.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.766+0000 7efe9cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:10:46.746 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:10:46.746 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:10:37.063243+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:10:37.063243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:10:46.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.771+0000 7efea5099640 10 librados: watch_flush enter 2024-03-29T11:10:46.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.771+0000 7efea5099640 10 librados: watch_flush exit 2024-03-29T11:10:46.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.771+0000 7efea5099640 20 client.4605.objecter shutdown clearing up homeless session... 2024-03-29T11:10:46.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.771+0000 7efea5099640 10 client.4605.objecter successfully canceled tick 2024-03-29T11:10:46.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.771+0000 7efea5099640 1 -- 172.21.15.71:0/2251826146 >> v1:172.21.15.137:6816/973332665 conn(0x7efe8406e080 legacy=0x7efe84070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:46.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.771+0000 7efea5099640 10 monclient: shutdown 2024-03-29T11:10:46.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.771+0000 7efea5099640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:46.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.771+0000 7efea5099640 1 -- 172.21.15.71:0/2251826146 >> v1:172.21.15.71:6789/0 conn(0x7efe980c92b0 legacy=0x7efe98160b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:46.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.771+0000 7efea5099640 1 -- 172.21.15.71:0/2251826146 shutdown_connections 2024-03-29T11:10:46.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.771+0000 7efea5099640 1 -- 172.21.15.71:0/2251826146 >> 172.21.15.71:0/2251826146 conn(0x7efe980058f0 msgr2=0x7efe98036a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:46.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.771+0000 7efea5099640 1 -- 172.21.15.71:0/2251826146 shutdown_connections 2024-03-29T11:10:46.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.772+0000 7efea5099640 1 -- 172.21.15.71:0/2251826146 wait complete. 2024-03-29T11:10:46.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:46.772+0000 7efea5099640 1 librados: shutdown 2024-03-29T11:10:46.766 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:10:53.573 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:10:53.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.758+0000 7f8c70cb5640 10 monclient: get_monmap_and_config 2024-03-29T11:10:53.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.758+0000 7f8c70cb5640 10 monclient: build_initial_monmap 2024-03-29T11:10:53.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.758+0000 7f8c70cb5640 10 monclient: monmap: 2024-03-29T11:10:53.736 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:10:53.736 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:53.736 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:10:53.758960+0000 2024-03-29T11:10:53.736 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:10:53.758960+0000 2024-03-29T11:10:53.736 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:53.736 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:53.737 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:10:53.737 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:10:53.737 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:10:53.737 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:53.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.760+0000 7f8c70cb5640 1 Processor -- start 2024-03-29T11:10:53.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.762+0000 7f8c70cb5640 1 -- start start 2024-03-29T11:10:53.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.762+0000 7f8c70cb5640 10 monclient: init 2024-03-29T11:10:53.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.763+0000 7f8c70cb5640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:53.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.763+0000 7f8c70cb5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:10:53.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.763+0000 7f8c70cb5640 10 monclient(hunting): picked mon.noname-a con 0x7f8c6c1422b0 addr v1:172.21.15.71:6789/0 2024-03-29T11:10:53.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.763+0000 7f8c70cb5640 10 monclient(hunting): picked mon.noname-b con 0x7f8c6c142690 addr v1:172.21.15.71:6790/0 2024-03-29T11:10:53.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.763+0000 7f8c70cb5640 10 monclient(hunting): picked mon.noname-c con 0x7f8c6c0b17d0 addr v1:172.21.15.137:6789/0 2024-03-29T11:10:53.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.763+0000 7f8c70cb5640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c6c142c80 con 0x7f8c6c1422b0 2024-03-29T11:10:53.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.763+0000 7f8c70cb5640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c6c0aae40 con 0x7f8c6c0b17d0 2024-03-29T11:10:53.742 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.763+0000 7f8c70cb5640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c6c13fbd0 con 0x7f8c6c142690 2024-03-29T11:10:53.743 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.763+0000 7f8c70cb5640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:53.743 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.763+0000 7f8c70cb5640 10 monclient(hunting): authenticate will time out at 1410.021569s 2024-03-29T11:10:53.743 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.763+0000 7f8c69d9b640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f8c6c142690 0x7f8c6c0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:56174/0 (socket says 172.21.15.71:56174) 2024-03-29T11:10:53.743 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.763+0000 7f8c69d9b640 1 -- 172.21.15.71:0/1284205021 learned_addr learned my addr 172.21.15.71:0/1284205021 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:10:53.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.767+0000 7f8c6959a640 1 -- 172.21.15.71:0/1284205021 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3931040277 0 0) 0x7f8c6c13fbd0 con 0x7f8c6c142690 2024-03-29T11:10:53.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.768+0000 7f8c6959a640 10 monclient(hunting): _negotiate 2024-03-29T11:10:53.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.768+0000 7f8c6959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:53.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.768+0000 7f8c6959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:53.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.768+0000 7f8c6959a640 10 monclient(hunting): my global_id is 4676 2024-03-29T11:10:53.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.768+0000 7f8c6959a640 1 -- 172.21.15.71:0/1284205021 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c58003520 con 0x7f8c6c142690 2024-03-29T11:10:53.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.768+0000 7f8c6959a640 1 -- 172.21.15.71:0/1284205021 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2579750468 0 0) 0x7f8c6c0aae40 con 0x7f8c6c0b17d0 2024-03-29T11:10:53.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.768+0000 7f8c6959a640 10 monclient(hunting): _negotiate 2024-03-29T11:10:53.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.768+0000 7f8c6959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:53.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.768+0000 7f8c6959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:53.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.768+0000 7f8c6959a640 10 monclient(hunting): my global_id is 4534 2024-03-29T11:10:53.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.768+0000 7f8c6959a640 1 -- 172.21.15.71:0/1284205021 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c6c13fbd0 con 0x7f8c6c0b17d0 2024-03-29T11:10:53.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.768+0000 7f8c6959a640 1 -- 172.21.15.71:0/1284205021 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675116934 0 0) 0x7f8c6c142c80 con 0x7f8c6c1422b0 2024-03-29T11:10:53.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.768+0000 7f8c6959a640 10 monclient(hunting): _negotiate 2024-03-29T11:10:53.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.768+0000 7f8c6959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:53.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.768+0000 7f8c6959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:53.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.769+0000 7f8c6959a640 10 monclient(hunting): my global_id is 4608 2024-03-29T11:10:53.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.769+0000 7f8c6959a640 1 -- 172.21.15.71:0/1284205021 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c6c0aae40 con 0x7f8c6c1422b0 2024-03-29T11:10:53.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.769+0000 7f8c6959a640 1 -- 172.21.15.71:0/1284205021 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1777546823 0 0) 0x7f8c6c13fbd0 con 0x7f8c6c0b17d0 2024-03-29T11:10:53.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.769+0000 7f8c6959a640 1 -- 172.21.15.71:0/1284205021 >> v1:172.21.15.71:6790/0 conn(0x7f8c6c142690 legacy=0x7f8c6c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:53.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.769+0000 7f8c6959a640 1 -- 172.21.15.71:0/1284205021 >> v1:172.21.15.71:6789/0 conn(0x7f8c6c1422b0 legacy=0x7f8c6c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:53.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.769+0000 7f8c6959a640 10 monclient: _finish_hunting 0 2024-03-29T11:10:53.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.769+0000 7f8c6959a640 1 monclient: found mon.noname-c 2024-03-29T11:10:53.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.769+0000 7f8c6959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:53.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.769+0000 7f8c6959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.137:6789/0 2024-03-29T11:10:53.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.770+0000 7f8c6959a640 1 -- 172.21.15.71:0/1284205021 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c6c151940 con 0x7f8c6c0b17d0 2024-03-29T11:10:53.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.770+0000 7f8c6959a640 10 monclient: _finish_auth 0 2024-03-29T11:10:53.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.770+0000 7f8c6959a640 10 monclient: _check_auth_tickets 2024-03-29T11:10:53.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.770+0000 7f8c6959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:53.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.770+0000 7f8c6959a640 1 -- 172.21.15.71:0/1284205021 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f8c54001670 con 0x7f8c6c0b17d0 2024-03-29T11:10:53.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.770+0000 7f8c70cb5640 5 monclient: authenticate success, global_id 4534 2024-03-29T11:10:53.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.770+0000 7f8c70cb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:53.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.770+0000 7f8c6959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:53.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.770+0000 7f8c6959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:10:53.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.770+0000 7f8c6959a640 10 monclient: dump: 2024-03-29T11:10:53.753 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:53.753 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:53.753 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:53.753 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:53.753 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:53.753 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:53.753 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:53.753 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:53.753 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:53.754 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:53.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.770+0000 7f8c70cb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:53.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.771+0000 7f8c6959a640 1 -- 172.21.15.71:0/1284205021 <== mon.1 v1:172.21.15.137:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c540017d0 con 0x7f8c6c0b17d0 2024-03-29T11:10:53.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.771+0000 7f8c6959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:53.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.771+0000 7f8c70cb5640 10 monclient: get_monmap_and_config success 2024-03-29T11:10:53.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.771+0000 7f8c70cb5640 4 set_mon_vals no callback set 2024-03-29T11:10:53.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.771+0000 7f8c6959a640 1 -- 172.21.15.71:0/1284205021 <== mon.1 v1:172.21.15.137:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f8c54001b60 con 0x7f8c6c0b17d0 2024-03-29T11:10:53.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.771+0000 7f8c6959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:53.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.771+0000 7f8c6959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:10:53.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.771+0000 7f8c6959a640 10 monclient: dump: 2024-03-29T11:10:53.754 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:53.754 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:53.754 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:53.754 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:53.755 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:53.755 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:53.755 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:53.755 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:53.755 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:53.755 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:53.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.772+0000 7f8c70cb5640 10 monclient: shutdown 2024-03-29T11:10:53.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.772+0000 7f8c70cb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:53.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.772+0000 7f8c70cb5640 1 -- 172.21.15.71:0/1284205021 >> v1:172.21.15.137:6789/0 conn(0x7f8c6c0b17d0 legacy=0x7f8c6c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:53.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.772+0000 7f8c70cb5640 1 -- 172.21.15.71:0/1284205021 shutdown_connections 2024-03-29T11:10:53.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.772+0000 7f8c70cb5640 1 -- 172.21.15.71:0/1284205021 >> 172.21.15.71:0/1284205021 conn(0x7f8c6c066a00 msgr2=0x7f8c6c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:53.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.774+0000 7f8c70cb5640 1 -- 172.21.15.71:0/1284205021 shutdown_connections 2024-03-29T11:10:53.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.774+0000 7f8c70cb5640 1 -- 172.21.15.71:0/1284205021 wait complete. 2024-03-29T11:10:53.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.775+0000 7f8c70cb5640 10 monclient: build_initial_monmap 2024-03-29T11:10:53.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.776+0000 7f8c70cb5640 10 monclient: monmap: 2024-03-29T11:10:53.758 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:10:53.758 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:10:53.758 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:10:53.758 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:10:53.759 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:53.759 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:53.759 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:10:53.759 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:10:53.759 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:10:53.759 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:53.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.776+0000 7f8c70cb5640 1 librados: starting msgr at 2024-03-29T11:10:53.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.776+0000 7f8c70cb5640 1 librados: starting objecter 2024-03-29T11:10:53.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.776+0000 7f8c70cb5640 1 Processor -- start 2024-03-29T11:10:53.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.776+0000 7f8c70cb5640 1 -- start start 2024-03-29T11:10:53.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.776+0000 7f8c70cb5640 1 librados: setting wanted keys 2024-03-29T11:10:53.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.776+0000 7f8c70cb5640 1 librados: calling monclient init 2024-03-29T11:10:53.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.776+0000 7f8c70cb5640 10 monclient: init 2024-03-29T11:10:53.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.777+0000 7f8c70cb5640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:53.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.777+0000 7f8c70cb5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:10:53.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.777+0000 7f8c70cb5640 10 monclient(hunting): picked mon.noname-a con 0x7f8c6c0b17d0 addr v1:172.21.15.71:6789/0 2024-03-29T11:10:53.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.777+0000 7f8c70cb5640 10 monclient(hunting): picked mon.noname-b con 0x7f8c6c1422b0 addr v1:172.21.15.137:6789/0 2024-03-29T11:10:53.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.777+0000 7f8c70cb5640 10 monclient(hunting): picked mon.noname-c con 0x7f8c6c142690 addr v1:172.21.15.71:6790/0 2024-03-29T11:10:53.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.777+0000 7f8c70cb5640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c6c06e590 con 0x7f8c6c0b17d0 2024-03-29T11:10:53.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.777+0000 7f8c70cb5640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c6c070980 con 0x7f8c6c1422b0 2024-03-29T11:10:53.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.777+0000 7f8c70cb5640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c6c06e7e0 con 0x7f8c6c142690 2024-03-29T11:10:53.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.777+0000 7f8c70cb5640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:53.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.777+0000 7f8c70cb5640 10 monclient(hunting): authenticate will time out at 1410.035926s 2024-03-29T11:10:53.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.778+0000 7f8c6ad9d640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f8c6c142690 0x7f8c6c06f410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:49776/0 (socket says 172.21.15.71:49776) 2024-03-29T11:10:53.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.778+0000 7f8c6ad9d640 1 -- 172.21.15.71:0/2107399548 learned_addr learned my addr 172.21.15.71:0/2107399548 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:10:53.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.779+0000 7f8c62ffd640 10 client.?.objecter ms_handle_connect 0x7f8c6c142690 2024-03-29T11:10:53.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.779+0000 7f8c62ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:53.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.781+0000 7f8c62ffd640 10 client.?.objecter ms_handle_connect 0x7f8c6c0b17d0 2024-03-29T11:10:53.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.781+0000 7f8c62ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:53.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.781+0000 7f8c62ffd640 1 -- 172.21.15.71:0/2107399548 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2960040062 0 0) 0x7f8c6c06e590 con 0x7f8c6c0b17d0 2024-03-29T11:10:53.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.781+0000 7f8c62ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:10:53.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.782+0000 7f8c62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:53.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.782+0000 7f8c62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:53.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.782+0000 7f8c62ffd640 10 monclient(hunting): my global_id is 4611 2024-03-29T11:10:53.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.782+0000 7f8c62ffd640 1 -- 172.21.15.71:0/2107399548 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c4c004eb0 con 0x7f8c6c0b17d0 2024-03-29T11:10:53.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.783+0000 7f8c62ffd640 1 -- 172.21.15.71:0/2107399548 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1273414126 0 0) 0x7f8c4c004eb0 con 0x7f8c6c0b17d0 2024-03-29T11:10:53.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.783+0000 7f8c62ffd640 1 -- 172.21.15.71:0/2107399548 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c6c06e590 con 0x7f8c6c0b17d0 2024-03-29T11:10:53.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.783+0000 7f8c62ffd640 1 -- 172.21.15.71:0/2107399548 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f8c5c001c70 con 0x7f8c6c0b17d0 2024-03-29T11:10:53.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.783+0000 7f8c62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:53.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.783+0000 7f8c62ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:10:53.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.784+0000 7f8c62ffd640 10 monclient(hunting): dump: 2024-03-29T11:10:53.763 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:53.763 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:53.763 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:53.763 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:53.763 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:53.763 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:53.763 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:53.764 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:53.764 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:53.764 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:53.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.784+0000 7f8c62ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:10:53.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.784+0000 7f8c62ffd640 1 -- 172.21.15.71:0/2107399548 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 576907202 0 0) 0x7f8c6c06e590 con 0x7f8c6c0b17d0 2024-03-29T11:10:53.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.785+0000 7f8c62ffd640 1 -- 172.21.15.71:0/2107399548 >> v1:172.21.15.71:6790/0 conn(0x7f8c6c142690 legacy=0x7f8c6c06f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:53.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.785+0000 7f8c62ffd640 1 -- 172.21.15.71:0/2107399548 >> v1:172.21.15.137:6789/0 conn(0x7f8c6c1422b0 legacy=0x7f8c6c06ed00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:10:53.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.785+0000 7f8c62ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:10:53.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.785+0000 7f8c62ffd640 1 monclient: found mon.a 2024-03-29T11:10:53.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.785+0000 7f8c62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:53.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.785+0000 7f8c62ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:10:53.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.785+0000 7f8c62ffd640 1 -- 172.21.15.71:0/2107399548 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c6c071b00 con 0x7f8c6c0b17d0 2024-03-29T11:10:53.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.785+0000 7f8c62ffd640 10 monclient: _finish_auth 0 2024-03-29T11:10:53.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.786+0000 7f8c62ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:10:53.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.786+0000 7f8c62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:53.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.786+0000 7f8c70cb5640 5 monclient: authenticate success, global_id 4611 2024-03-29T11:10:53.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.786+0000 7f8c70cb5640 10 monclient: _renew_subs 2024-03-29T11:10:53.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.786+0000 7f8c70cb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:10:53.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.786+0000 7f8c70cb5640 1 -- 172.21.15.71:0/2107399548 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c6c071d50 con 0x7f8c6c0b17d0 2024-03-29T11:10:53.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.786+0000 7f8c70cb5640 10 client.4611.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:10:53.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.786+0000 7f8c70cb5640 10 monclient: _renew_subs 2024-03-29T11:10:53.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.786+0000 7f8c70cb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:10:53.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.786+0000 7f8c70cb5640 1 -- 172.21.15.71:0/2107399548 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c6c1e7280 con 0x7f8c6c0b17d0 2024-03-29T11:10:53.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.786+0000 7f8c70cb5640 1 librados: init done 2024-03-29T11:10:53.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.786+0000 7f8c62ffd640 1 -- 172.21.15.71:0/2107399548 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c5c003990 con 0x7f8c6c0b17d0 2024-03-29T11:10:53.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.786+0000 7f8c62ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:53.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.786+0000 7f8c62ffd640 1 -- 172.21.15.71:0/2107399548 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f8c5c003da0 con 0x7f8c6c0b17d0 2024-03-29T11:10:53.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.786+0000 7f8c62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:53.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.786+0000 7f8c62ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:10:53.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.786+0000 7f8c62ffd640 10 monclient: dump: 2024-03-29T11:10:53.770 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:10:53.771 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:10:53.771 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:53.771 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:10:53.771 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:53.771 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:10:53.771 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:10:53.771 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:10:53.771 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:10:53.771 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:10:53.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.786+0000 7f8c68d99640 4 set_mon_vals no callback set 2024-03-29T11:10:53.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.789+0000 7f8c62ffd640 1 -- 172.21.15.71:0/2107399548 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f8c5c003f00 con 0x7f8c6c0b17d0 2024-03-29T11:10:53.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.790+0000 7f8c62ffd640 1 -- 172.21.15.71:0/2107399548 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 45029510 0 0) 0x7f8c5c083c80 con 0x7f8c6c0b17d0 2024-03-29T11:10:53.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.790+0000 7f8c62ffd640 10 client.4611.objecter ms_dispatch 0x7f8c6c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:10:53.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.790+0000 7f8c62ffd640 3 client.4611.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:10:53.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.790+0000 7f8c62ffd640 3 client.4611.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:10:53.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.790+0000 7f8c62ffd640 20 client.4611.objecter dump_active .. 0 homeless 2024-03-29T11:10:53.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.793+0000 7f8c69d9b640 10 monclient: get_auth_request con 0x7f8c4c06dd50 auth_method 0 2024-03-29T11:10:53.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.793+0000 7f8c70cb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:53.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.793+0000 7f8c62ffd640 10 client.4611.objecter ms_handle_connect 0x7f8c4c06dd50 2024-03-29T11:10:53.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.794+0000 7f8c70cb5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:53.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.794+0000 7f8c70cb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:10:53.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.794+0000 7f8c70cb5640 1 -- 172.21.15.71:0/2107399548 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c6c075f10 con 0x7f8c6c0b17d0 2024-03-29T11:10:53.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.804+0000 7f8c62ffd640 1 -- 172.21.15.71:0/2107399548 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8c5c083e60 con 0x7f8c6c0b17d0 2024-03-29T11:10:53.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.804+0000 7f8c62ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:53.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:53.804+0000 7f8c62ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:10:54.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:54.148+0000 7f8c70cb5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:54.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:54.148+0000 7f8c70cb5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:54.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:54.148+0000 7f8c70cb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:10:54.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:54.148+0000 7f8c70cb5640 1 -- 172.21.15.71:0/2107399548 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c6c074f10 con 0x7f8c6c0b17d0 2024-03-29T11:10:54.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:54.150+0000 7f8c62ffd640 1 -- 172.21.15.71:0/2107399548 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 577002532) 0x7f8c5c04f390 con 0x7f8c6c0b17d0 2024-03-29T11:10:54.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:54.150+0000 7f8c62ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:54.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:54.150+0000 7f8c62ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:10:54.128 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:10:54.129 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:10:37.063243+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:10:37.063243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:10:54.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:54.153+0000 7f8c70cb5640 10 librados: watch_flush enter 2024-03-29T11:10:54.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:54.156+0000 7f8c70cb5640 10 librados: watch_flush exit 2024-03-29T11:10:54.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:54.156+0000 7f8c70cb5640 20 client.4611.objecter shutdown clearing up homeless session... 2024-03-29T11:10:54.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:54.156+0000 7f8c70cb5640 10 client.4611.objecter successfully canceled tick 2024-03-29T11:10:54.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:54.156+0000 7f8c70cb5640 1 -- 172.21.15.71:0/2107399548 >> v1:172.21.15.137:6816/973332665 conn(0x7f8c4c06dd50 legacy=0x7f8c4c070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:54.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:54.156+0000 7f8c70cb5640 10 monclient: shutdown 2024-03-29T11:10:54.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:54.156+0000 7f8c70cb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:54.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:54.156+0000 7f8c70cb5640 1 -- 172.21.15.71:0/2107399548 >> v1:172.21.15.71:6789/0 conn(0x7f8c6c0b17d0 legacy=0x7f8c6c0713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:54.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:54.158+0000 7f8c70cb5640 1 -- 172.21.15.71:0/2107399548 shutdown_connections 2024-03-29T11:10:54.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:54.158+0000 7f8c70cb5640 1 -- 172.21.15.71:0/2107399548 >> 172.21.15.71:0/2107399548 conn(0x7f8c6c066a00 msgr2=0x7f8c6c0b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:54.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:54.160+0000 7f8c70cb5640 1 -- 172.21.15.71:0/2107399548 shutdown_connections 2024-03-29T11:10:54.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:54.160+0000 7f8c70cb5640 1 -- 172.21.15.71:0/2107399548 wait complete. 2024-03-29T11:10:54.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:10:54.160+0000 7f8c70cb5640 1 librados: shutdown 2024-03-29T11:10:54.155 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:11:00.963 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:01.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.130+0000 7f3c6a915640 10 monclient: get_monmap_and_config 2024-03-29T11:11:01.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.130+0000 7f3c6a915640 10 monclient: build_initial_monmap 2024-03-29T11:11:01.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.130+0000 7f3c6a915640 10 monclient: monmap: 2024-03-29T11:11:01.109 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:11:01.109 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:01.109 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:11:01.131556+0000 2024-03-29T11:11:01.109 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:11:01.131556+0000 2024-03-29T11:11:01.109 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:01.109 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:01.109 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:11:01.109 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:11:01.109 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:11:01.109 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:01.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.131+0000 7f3c6a915640 1 Processor -- start 2024-03-29T11:11:01.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.131+0000 7f3c6a915640 1 -- start start 2024-03-29T11:11:01.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.131+0000 7f3c6a915640 10 monclient: init 2024-03-29T11:11:01.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.132+0000 7f3c6a915640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:01.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.132+0000 7f3c6a915640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:11:01.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.135+0000 7f3c6a915640 10 monclient(hunting): picked mon.noname-a con 0x7f3c640b4330 addr v1:172.21.15.71:6789/0 2024-03-29T11:11:01.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.136+0000 7f3c6a915640 10 monclient(hunting): picked mon.noname-c con 0x7f3c641470b0 addr v1:172.21.15.137:6789/0 2024-03-29T11:11:01.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.136+0000 7f3c6a915640 10 monclient(hunting): picked mon.noname-b con 0x7f3c641439b0 addr v1:172.21.15.71:6790/0 2024-03-29T11:11:01.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.136+0000 7f3c6a915640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c6406f2a0 con 0x7f3c640b4330 2024-03-29T11:11:01.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.136+0000 7f3c6a915640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c64146b90 con 0x7f3c641470b0 2024-03-29T11:11:01.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.136+0000 7f3c6a915640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c640b2ca0 con 0x7f3c641439b0 2024-03-29T11:11:01.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.136+0000 7f3c6a915640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:01.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.136+0000 7f3c6a915640 10 monclient(hunting): authenticate will time out at 1417.394218s 2024-03-29T11:11:01.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.136+0000 7f3c63fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f3c640b4330 0x7f3c640b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:36339/0 (socket says 172.21.15.71:36339) 2024-03-29T11:11:01.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.136+0000 7f3c63fff640 1 -- 172.21.15.71:0/3750451843 learned_addr learned my addr 172.21.15.71:0/3750451843 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:11:01.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.136+0000 7f3c62ffd640 1 -- 172.21.15.71:0/3750451843 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186719354 0 0) 0x7f3c6406f2a0 con 0x7f3c640b4330 2024-03-29T11:11:01.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.136+0000 7f3c62ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:11:01.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.136+0000 7f3c62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:01.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.136+0000 7f3c62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:01.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.136+0000 7f3c62ffd640 10 monclient(hunting): my global_id is 4614 2024-03-29T11:11:01.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.136+0000 7f3c62ffd640 1 -- 172.21.15.71:0/3750451843 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c4c003520 con 0x7f3c640b4330 2024-03-29T11:11:01.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.137+0000 7f3c62ffd640 1 -- 172.21.15.71:0/3750451843 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2748504625 0 0) 0x7f3c4c003520 con 0x7f3c640b4330 2024-03-29T11:11:01.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.137+0000 7f3c62ffd640 1 -- 172.21.15.71:0/3750451843 >> v1:172.21.15.71:6790/0 conn(0x7f3c641439b0 legacy=0x7f3c64151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:01.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.137+0000 7f3c62ffd640 1 -- 172.21.15.71:0/3750451843 >> v1:172.21.15.137:6789/0 conn(0x7f3c641470b0 legacy=0x7f3c640b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:11:01.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.138+0000 7f3c62ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:11:01.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.138+0000 7f3c62ffd640 1 monclient: found mon.noname-a 2024-03-29T11:11:01.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.138+0000 7f3c62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:01.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.138+0000 7f3c62ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:11:01.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.138+0000 7f3c62ffd640 1 -- 172.21.15.71:0/3750451843 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6406e0c0 con 0x7f3c640b4330 2024-03-29T11:11:01.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.138+0000 7f3c62ffd640 10 monclient: _finish_auth 0 2024-03-29T11:11:01.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.138+0000 7f3c62ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:11:01.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.138+0000 7f3c62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:01.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.138+0000 7f3c62ffd640 1 -- 172.21.15.71:0/3750451843 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f3c5c001a10 con 0x7f3c640b4330 2024-03-29T11:11:01.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.138+0000 7f3c6a915640 5 monclient: authenticate success, global_id 4614 2024-03-29T11:11:01.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.138+0000 7f3c6a915640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:01.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.138+0000 7f3c62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:01.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.138+0000 7f3c62ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:11:01.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.138+0000 7f3c62ffd640 10 monclient: dump: 2024-03-29T11:11:01.117 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:11:01.117 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:01.117 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:01.118 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:01.118 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:01.118 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:01.118 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:11:01.118 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:11:01.118 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:11:01.118 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:01.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.138+0000 7f3c6a915640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:01.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.138+0000 7f3c62ffd640 1 -- 172.21.15.71:0/3750451843 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c5c001ca0 con 0x7f3c640b4330 2024-03-29T11:11:01.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.138+0000 7f3c62ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:01.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.138+0000 7f3c62ffd640 1 -- 172.21.15.71:0/3750451843 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f3c5c002030 con 0x7f3c640b4330 2024-03-29T11:11:01.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.138+0000 7f3c62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:01.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.138+0000 7f3c62ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:11:01.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.138+0000 7f3c62ffd640 10 monclient: dump: 2024-03-29T11:11:01.119 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:11:01.119 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:01.119 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:01.119 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:01.119 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:01.119 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:01.119 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:11:01.119 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:11:01.120 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:11:01.120 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:01.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.138+0000 7f3c6a915640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:01.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.138+0000 7f3c6a915640 4 set_mon_vals no callback set 2024-03-29T11:11:01.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.139+0000 7f3c6a915640 10 monclient: shutdown 2024-03-29T11:11:01.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.139+0000 7f3c6a915640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:01.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.139+0000 7f3c6a915640 1 -- 172.21.15.71:0/3750451843 >> v1:172.21.15.71:6789/0 conn(0x7f3c640b4330 legacy=0x7f3c640b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:01.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.141+0000 7f3c6a915640 1 -- 172.21.15.71:0/3750451843 shutdown_connections 2024-03-29T11:11:01.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.141+0000 7f3c6a915640 1 -- 172.21.15.71:0/3750451843 >> 172.21.15.71:0/3750451843 conn(0x7f3c64066a00 msgr2=0x7f3c64066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:01.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.142+0000 7f3c6a915640 1 -- 172.21.15.71:0/3750451843 shutdown_connections 2024-03-29T11:11:01.123 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.145+0000 7f3c6a915640 1 -- 172.21.15.71:0/3750451843 wait complete. 2024-03-29T11:11:01.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.146+0000 7f3c6a915640 10 monclient: build_initial_monmap 2024-03-29T11:11:01.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.146+0000 7f3c6a915640 10 monclient: monmap: 2024-03-29T11:11:01.124 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:11:01.124 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:01.124 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:11:01.124 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:11:01.124 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:01.124 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:01.124 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:11:01.124 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:11:01.124 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:11:01.124 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:01.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.146+0000 7f3c6a915640 1 librados: starting msgr at 2024-03-29T11:11:01.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.146+0000 7f3c6a915640 1 librados: starting objecter 2024-03-29T11:11:01.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.146+0000 7f3c6a915640 1 Processor -- start 2024-03-29T11:11:01.126 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.147+0000 7f3c6a915640 1 -- start start 2024-03-29T11:11:01.126 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.147+0000 7f3c6a915640 1 librados: setting wanted keys 2024-03-29T11:11:01.126 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.147+0000 7f3c6a915640 1 librados: calling monclient init 2024-03-29T11:11:01.126 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.147+0000 7f3c6a915640 10 monclient: init 2024-03-29T11:11:01.126 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.148+0000 7f3c6a915640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:01.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.148+0000 7f3c6a915640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:11:01.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.148+0000 7f3c6a915640 10 monclient(hunting): picked mon.noname-a con 0x7f3c640b4330 addr v1:172.21.15.71:6789/0 2024-03-29T11:11:01.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.148+0000 7f3c6a915640 10 monclient(hunting): picked mon.noname-c con 0x7f3c641439b0 addr v1:172.21.15.71:6790/0 2024-03-29T11:11:01.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.150+0000 7f3c637fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f3c641439b0 0x7f3c64140b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:59598/0 (socket says 172.21.15.71:59598) 2024-03-29T11:11:01.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.152+0000 7f3c637fe640 1 -- 172.21.15.71:0/548756937 learned_addr learned my addr 172.21.15.71:0/548756937 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:11:01.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.152+0000 7f3c6a915640 10 monclient(hunting): picked mon.noname-b con 0x7f3c641470b0 addr v1:172.21.15.137:6789/0 2024-03-29T11:11:01.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.152+0000 7f3c6a915640 1 -- 172.21.15.71:0/548756937 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c64120ec0 con 0x7f3c640b4330 2024-03-29T11:11:01.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.152+0000 7f3c60ff9640 10 client.?.objecter ms_handle_connect 0x7f3c641439b0 2024-03-29T11:11:01.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.152+0000 7f3c60ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:01.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.152+0000 7f3c6a915640 1 -- 172.21.15.71:0/548756937 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c64141940 con 0x7f3c641470b0 2024-03-29T11:11:01.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.152+0000 7f3c6a915640 1 -- 172.21.15.71:0/548756937 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c64121130 con 0x7f3c641439b0 2024-03-29T11:11:01.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.152+0000 7f3c6a915640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:01.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.152+0000 7f3c6a915640 10 monclient(hunting): authenticate will time out at 1417.410860s 2024-03-29T11:11:01.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.152+0000 7f3c60ff9640 1 -- 172.21.15.71:0/548756937 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3992990915 0 0) 0x7f3c64121130 con 0x7f3c641439b0 2024-03-29T11:11:01.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.152+0000 7f3c60ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:11:01.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.153+0000 7f3c60ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:01.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.153+0000 7f3c60ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:01.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.153+0000 7f3c60ff9640 10 monclient(hunting): my global_id is 4685 2024-03-29T11:11:01.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.153+0000 7f3c60ff9640 1 -- 172.21.15.71:0/548756937 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c50004eb0 con 0x7f3c641439b0 2024-03-29T11:11:01.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.153+0000 7f3c60ff9640 10 client.?.objecter ms_handle_connect 0x7f3c641470b0 2024-03-29T11:11:01.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.153+0000 7f3c60ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:01.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.153+0000 7f3c60ff9640 10 client.?.objecter ms_handle_connect 0x7f3c640b4330 2024-03-29T11:11:01.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.153+0000 7f3c60ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:01.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.153+0000 7f3c60ff9640 1 -- 172.21.15.71:0/548756937 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1642030218 0 0) 0x7f3c64120ec0 con 0x7f3c640b4330 2024-03-29T11:11:01.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.153+0000 7f3c60ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:11:01.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.153+0000 7f3c60ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:01.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.153+0000 7f3c60ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:01.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.153+0000 7f3c60ff9640 10 monclient(hunting): my global_id is 4617 2024-03-29T11:11:01.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.154+0000 7f3c60ff9640 1 -- 172.21.15.71:0/548756937 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c64121130 con 0x7f3c640b4330 2024-03-29T11:11:01.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.154+0000 7f3c60ff9640 1 -- 172.21.15.71:0/548756937 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3061977818 0 0) 0x7f3c64141940 con 0x7f3c641470b0 2024-03-29T11:11:01.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.154+0000 7f3c60ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:11:01.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.154+0000 7f3c60ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:01.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.154+0000 7f3c60ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:01.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.154+0000 7f3c60ff9640 10 monclient(hunting): my global_id is 4537 2024-03-29T11:11:01.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.154+0000 7f3c60ff9640 1 -- 172.21.15.71:0/548756937 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c64120ec0 con 0x7f3c641470b0 2024-03-29T11:11:01.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.154+0000 7f3c60ff9640 1 -- 172.21.15.71:0/548756937 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4121097643 0 0) 0x7f3c50004eb0 con 0x7f3c641439b0 2024-03-29T11:11:01.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.154+0000 7f3c60ff9640 1 -- 172.21.15.71:0/548756937 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c64141940 con 0x7f3c641439b0 2024-03-29T11:11:01.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.154+0000 7f3c60ff9640 1 -- 172.21.15.71:0/548756937 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f3c540017f0 con 0x7f3c641439b0 2024-03-29T11:11:01.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.154+0000 7f3c60ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:01.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.154+0000 7f3c60ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:11:01.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.154+0000 7f3c60ff9640 10 monclient(hunting): dump: 2024-03-29T11:11:01.140 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:11:01.140 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:01.140 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:01.140 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:01.140 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:01.140 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:01.140 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:11:01.141 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:11:01.141 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:11:01.141 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:01.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.154+0000 7f3c60ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:01.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.154+0000 7f3c60ff9640 1 -- 172.21.15.71:0/548756937 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2712705032 0 0) 0x7f3c64121130 con 0x7f3c640b4330 2024-03-29T11:11:01.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.154+0000 7f3c60ff9640 1 -- 172.21.15.71:0/548756937 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c50004eb0 con 0x7f3c640b4330 2024-03-29T11:11:01.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.154+0000 7f3c60ff9640 1 -- 172.21.15.71:0/548756937 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1243129000 0 0) 0x7f3c64120ec0 con 0x7f3c641470b0 2024-03-29T11:11:01.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.155+0000 7f3c60ff9640 1 -- 172.21.15.71:0/548756937 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c64121130 con 0x7f3c641470b0 2024-03-29T11:11:01.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.155+0000 7f3c60ff9640 1 -- 172.21.15.71:0/548756937 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f3c5c0018d0 con 0x7f3c640b4330 2024-03-29T11:11:01.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.155+0000 7f3c60ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:01.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.155+0000 7f3c60ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:11:01.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.155+0000 7f3c60ff9640 10 monclient(hunting): dump: 2024-03-29T11:11:01.141 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:11:01.141 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:01.142 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:01.142 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:01.142 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:01.142 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:01.142 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:11:01.142 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:11:01.142 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:11:01.142 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:01.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.155+0000 7f3c60ff9640 1 -- 172.21.15.71:0/548756937 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2013906939 0 0) 0x7f3c64141940 con 0x7f3c641439b0 2024-03-29T11:11:01.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.155+0000 7f3c60ff9640 1 -- 172.21.15.71:0/548756937 >> v1:172.21.15.137:6789/0 conn(0x7f3c641470b0 legacy=0x7f3c64141230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:01.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.155+0000 7f3c60ff9640 1 -- 172.21.15.71:0/548756937 >> v1:172.21.15.71:6789/0 conn(0x7f3c640b4330 legacy=0x7f3c6411fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:01.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.155+0000 7f3c60ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:11:01.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.155+0000 7f3c60ff9640 1 monclient: found mon.c 2024-03-29T11:11:01.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.155+0000 7f3c60ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:01.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.155+0000 7f3c60ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:11:01.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.155+0000 7f3c60ff9640 1 -- 172.21.15.71:0/548756937 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c64121410 con 0x7f3c641439b0 2024-03-29T11:11:01.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.155+0000 7f3c60ff9640 10 monclient: _finish_auth 0 2024-03-29T11:11:01.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.155+0000 7f3c60ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:11:01.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.156+0000 7f3c60ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:01.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.156+0000 7f3c6a915640 5 monclient: authenticate success, global_id 4685 2024-03-29T11:11:01.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.156+0000 7f3c6a915640 10 monclient: _renew_subs 2024-03-29T11:11:01.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.156+0000 7f3c6a915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:11:01.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.156+0000 7f3c6a915640 1 -- 172.21.15.71:0/548756937 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c6411f810 con 0x7f3c641439b0 2024-03-29T11:11:01.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.156+0000 7f3c6a915640 10 client.4685.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:01.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.156+0000 7f3c6a915640 10 monclient: _renew_subs 2024-03-29T11:11:01.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.156+0000 7f3c6a915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:11:01.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.156+0000 7f3c6a915640 1 -- 172.21.15.71:0/548756937 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c64141e00 con 0x7f3c641439b0 2024-03-29T11:11:01.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.156+0000 7f3c6a915640 1 librados: init done 2024-03-29T11:11:01.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.158+0000 7f3c60ff9640 1 -- 172.21.15.71:0/548756937 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c54001950 con 0x7f3c641439b0 2024-03-29T11:11:01.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.158+0000 7f3c60ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:01.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.158+0000 7f3c60ff9640 1 -- 172.21.15.71:0/548756937 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f3c54001e00 con 0x7f3c641439b0 2024-03-29T11:11:01.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.158+0000 7f3c627fc640 4 set_mon_vals no callback set 2024-03-29T11:11:01.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.158+0000 7f3c60ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:01.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.158+0000 7f3c60ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:01.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.158+0000 7f3c60ff9640 10 monclient: dump: 2024-03-29T11:11:01.144 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:11:01.144 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:01.144 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:01.144 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:01.144 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:01.144 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:01.144 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:11:01.144 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:11:01.145 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:11:01.145 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:01.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.159+0000 7f3c60ff9640 1 -- 172.21.15.71:0/548756937 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f3c5401a9d0 con 0x7f3c641439b0 2024-03-29T11:11:01.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.160+0000 7f3c60ff9640 1 -- 172.21.15.71:0/548756937 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 45029510 0 0) 0x7f3c54084930 con 0x7f3c641439b0 2024-03-29T11:11:01.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.160+0000 7f3c63fff640 10 monclient: get_auth_request con 0x7f3c5006e250 auth_method 0 2024-03-29T11:11:01.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.160+0000 7f3c6a915640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:01.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.160+0000 7f3c6a915640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:01.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.160+0000 7f3c6a915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:11:01.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.160+0000 7f3c6a915640 1 -- 172.21.15.71:0/548756937 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c44005080 con 0x7f3c641439b0 2024-03-29T11:11:01.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.160+0000 7f3c60ff9640 10 client.4685.objecter ms_dispatch 0x7f3c64000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:11:01.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.160+0000 7f3c60ff9640 3 client.4685.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:11:01.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.161+0000 7f3c60ff9640 3 client.4685.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:11:01.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.161+0000 7f3c60ff9640 20 client.4685.objecter dump_active .. 0 homeless 2024-03-29T11:11:01.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.161+0000 7f3c60ff9640 10 client.4685.objecter ms_handle_connect 0x7f3c5006e250 2024-03-29T11:11:01.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.170+0000 7f3c60ff9640 1 -- 172.21.15.71:0/548756937 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3c54050040 con 0x7f3c641439b0 2024-03-29T11:11:01.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.170+0000 7f3c60ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:01.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.170+0000 7f3c60ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:01.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.491+0000 7f3c6a915640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:01.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.491+0000 7f3c6a915640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:01.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.491+0000 7f3c6a915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:11:01.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.491+0000 7f3c6a915640 1 -- 172.21.15.71:0/548756937 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c44005330 con 0x7f3c641439b0 2024-03-29T11:11:01.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.492+0000 7f3c60ff9640 1 -- 172.21.15.71:0/548756937 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 577002532) 0x7f3c54052140 con 0x7f3c641439b0 2024-03-29T11:11:01.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.492+0000 7f3c60ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:01.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.492+0000 7f3c60ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:01.471 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:11:01.471 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:10:37.063243+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:10:37.063243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:01.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.496+0000 7f3c6a915640 10 librados: watch_flush enter 2024-03-29T11:11:01.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.496+0000 7f3c6a915640 10 librados: watch_flush exit 2024-03-29T11:11:01.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.496+0000 7f3c6a915640 20 client.4685.objecter shutdown clearing up homeless session... 2024-03-29T11:11:01.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.496+0000 7f3c6a915640 10 client.4685.objecter successfully canceled tick 2024-03-29T11:11:01.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.496+0000 7f3c6a915640 1 -- 172.21.15.71:0/548756937 >> v1:172.21.15.137:6816/973332665 conn(0x7f3c5006e250 legacy=0x7f3c50070710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:01.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.496+0000 7f3c6a915640 10 monclient: shutdown 2024-03-29T11:11:01.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.496+0000 7f3c6a915640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:01.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.496+0000 7f3c6a915640 1 -- 172.21.15.71:0/548756937 >> v1:172.21.15.71:6790/0 conn(0x7f3c641439b0 legacy=0x7f3c64140b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:01.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.496+0000 7f3c6a915640 1 -- 172.21.15.71:0/548756937 shutdown_connections 2024-03-29T11:11:01.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.496+0000 7f3c6a915640 1 -- 172.21.15.71:0/548756937 >> 172.21.15.71:0/548756937 conn(0x7f3c64066a00 msgr2=0x7f3c640b4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:01.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.497+0000 7f3c6a915640 1 -- 172.21.15.71:0/548756937 shutdown_connections 2024-03-29T11:11:01.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.498+0000 7f3c6a915640 1 -- 172.21.15.71:0/548756937 wait complete. 2024-03-29T11:11:01.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:01.498+0000 7f3c6a915640 1 librados: shutdown 2024-03-29T11:11:01.494 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:11:08.300 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:08.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.455+0000 7fb84ef79640 10 monclient: get_monmap_and_config 2024-03-29T11:11:08.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.456+0000 7fb84ef79640 10 monclient: build_initial_monmap 2024-03-29T11:11:08.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.456+0000 7fb84ef79640 10 monclient: monmap: 2024-03-29T11:11:08.435 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:11:08.435 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:08.436 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:11:08.457892+0000 2024-03-29T11:11:08.436 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:11:08.457892+0000 2024-03-29T11:11:08.436 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:08.436 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:08.436 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:11:08.436 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:11:08.436 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:11:08.436 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:08.437 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.456+0000 7fb84ef79640 1 Processor -- start 2024-03-29T11:11:08.437 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.457+0000 7fb84ef79640 1 -- start start 2024-03-29T11:11:08.437 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.457+0000 7fb84ef79640 10 monclient: init 2024-03-29T11:11:08.439 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.458+0000 7fb84ef79640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:08.439 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.458+0000 7fb84ef79640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:11:08.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.458+0000 7fb84ef79640 10 monclient(hunting): picked mon.noname-a con 0x7fb840006ec0 addr v1:172.21.15.71:6789/0 2024-03-29T11:11:08.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.458+0000 7fb84ef79640 10 monclient(hunting): picked mon.noname-c con 0x7fb840033530 addr v1:172.21.15.137:6789/0 2024-03-29T11:11:08.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.458+0000 7fb84ef79640 10 monclient(hunting): picked mon.noname-b con 0x7fb8400cfcb0 addr v1:172.21.15.71:6790/0 2024-03-29T11:11:08.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.458+0000 7fb84ef79640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8400cf110 con 0x7fb840006ec0 2024-03-29T11:11:08.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.458+0000 7fb84ef79640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb84002c1d0 con 0x7fb840033530 2024-03-29T11:11:08.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.458+0000 7fb84ef79640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8400cbb90 con 0x7fb8400cfcb0 2024-03-29T11:11:08.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.459+0000 7fb84ef79640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:08.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.459+0000 7fb84ef79640 10 monclient(hunting): authenticate will time out at 1424.718118s 2024-03-29T11:11:08.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.459+0000 7fb84df77640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fb840006ec0 0x7fb840032e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:60888/0 (socket says 172.21.15.71:60888) 2024-03-29T11:11:08.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.459+0000 7fb84df77640 1 -- 172.21.15.71:0/1827539239 learned_addr learned my addr 172.21.15.71:0/1827539239 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:11:08.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.459+0000 7fb84cf75640 1 -- 172.21.15.71:0/1827539239 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3858248835 0 0) 0x7fb8400cf110 con 0x7fb840006ec0 2024-03-29T11:11:08.440 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.459+0000 7fb84cf75640 10 monclient(hunting): _negotiate 2024-03-29T11:11:08.441 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.459+0000 7fb84cf75640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:08.441 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.459+0000 7fb84cf75640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:08.441 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.459+0000 7fb84cf75640 10 monclient(hunting): my global_id is 4620 2024-03-29T11:11:08.441 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.459+0000 7fb84cf75640 1 -- 172.21.15.71:0/1827539239 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb834003520 con 0x7fb840006ec0 2024-03-29T11:11:08.441 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.459+0000 7fb84cf75640 1 -- 172.21.15.71:0/1827539239 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 819769043 0 0) 0x7fb8400cbb90 con 0x7fb8400cfcb0 2024-03-29T11:11:08.441 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.459+0000 7fb84cf75640 10 monclient(hunting): _negotiate 2024-03-29T11:11:08.441 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.459+0000 7fb84cf75640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:08.441 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.459+0000 7fb84cf75640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:08.441 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.459+0000 7fb84cf75640 10 monclient(hunting): my global_id is 4688 2024-03-29T11:11:08.441 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.459+0000 7fb84cf75640 1 -- 172.21.15.71:0/1827539239 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8400cf110 con 0x7fb8400cfcb0 2024-03-29T11:11:08.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.459+0000 7fb84cf75640 1 -- 172.21.15.71:0/1827539239 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1801145982 0 0) 0x7fb834003520 con 0x7fb840006ec0 2024-03-29T11:11:08.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.459+0000 7fb84cf75640 1 -- 172.21.15.71:0/1827539239 >> v1:172.21.15.71:6790/0 conn(0x7fb8400cfcb0 legacy=0x7fb8400d74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:08.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.459+0000 7fb84cf75640 1 -- 172.21.15.71:0/1827539239 >> v1:172.21.15.137:6789/0 conn(0x7fb840033530 legacy=0x7fb8400cf5a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:11:08.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.461+0000 7fb84cf75640 10 monclient: _finish_hunting 0 2024-03-29T11:11:08.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.461+0000 7fb84cf75640 1 monclient: found mon.noname-a 2024-03-29T11:11:08.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.461+0000 7fb84cf75640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:08.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.461+0000 7fb84cf75640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:11:08.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.461+0000 7fb84cf75640 1 -- 172.21.15.71:0/1827539239 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb840029c30 con 0x7fb840006ec0 2024-03-29T11:11:08.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.461+0000 7fb84cf75640 10 monclient: _finish_auth 0 2024-03-29T11:11:08.452 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.461+0000 7fb84cf75640 10 monclient: _check_auth_tickets 2024-03-29T11:11:08.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.461+0000 7fb84cf75640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:08.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.461+0000 7fb84cf75640 1 -- 172.21.15.71:0/1827539239 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb844001690 con 0x7fb840006ec0 2024-03-29T11:11:08.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.461+0000 7fb84cf75640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:08.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.461+0000 7fb84cf75640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:11:08.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.461+0000 7fb84cf75640 10 monclient: dump: 2024-03-29T11:11:08.453 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:11:08.453 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:08.453 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:08.453 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:08.454 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:08.454 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:08.454 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:11:08.454 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:11:08.454 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:11:08.454 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:08.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.461+0000 7fb84ef79640 5 monclient: authenticate success, global_id 4620 2024-03-29T11:11:08.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.461+0000 7fb84ef79640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:08.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.461+0000 7fb84cf75640 1 -- 172.21.15.71:0/1827539239 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8440017f0 con 0x7fb840006ec0 2024-03-29T11:11:08.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.461+0000 7fb84cf75640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:08.455 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.461+0000 7fb84cf75640 1 -- 172.21.15.71:0/1827539239 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb844001c40 con 0x7fb840006ec0 2024-03-29T11:11:08.455 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.461+0000 7fb84cf75640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:08.455 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.461+0000 7fb84cf75640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:11:08.455 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.461+0000 7fb84cf75640 10 monclient: dump: 2024-03-29T11:11:08.455 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:11:08.455 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:08.455 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:08.455 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:08.455 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:08.455 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:08.455 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:11:08.455 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:11:08.456 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:11:08.456 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:08.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.461+0000 7fb84ef79640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:08.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.461+0000 7fb84ef79640 4 set_mon_vals no callback set 2024-03-29T11:11:08.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.463+0000 7fb84ef79640 10 monclient: shutdown 2024-03-29T11:11:08.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.463+0000 7fb84ef79640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:08.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.463+0000 7fb84ef79640 1 -- 172.21.15.71:0/1827539239 >> v1:172.21.15.71:6789/0 conn(0x7fb840006ec0 legacy=0x7fb840032e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:08.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.463+0000 7fb84ef79640 1 -- 172.21.15.71:0/1827539239 shutdown_connections 2024-03-29T11:11:08.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.463+0000 7fb84ef79640 1 -- 172.21.15.71:0/1827539239 >> 172.21.15.71:0/1827539239 conn(0x7fb840005d40 msgr2=0x7fb840006150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:08.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.463+0000 7fb84ef79640 1 -- 172.21.15.71:0/1827539239 shutdown_connections 2024-03-29T11:11:08.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.463+0000 7fb84ef79640 1 -- 172.21.15.71:0/1827539239 wait complete. 2024-03-29T11:11:08.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.464+0000 7fb84ef79640 10 monclient: build_initial_monmap 2024-03-29T11:11:08.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.464+0000 7fb84ef79640 10 monclient: monmap: 2024-03-29T11:11:08.457 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:11:08.457 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:08.457 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:11:08.457 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:11:08.457 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:08.457 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:08.457 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:11:08.457 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:11:08.457 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:11:08.457 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:08.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.464+0000 7fb84ef79640 1 librados: starting msgr at 2024-03-29T11:11:08.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.464+0000 7fb84ef79640 1 librados: starting objecter 2024-03-29T11:11:08.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.464+0000 7fb84ef79640 1 Processor -- start 2024-03-29T11:11:08.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.464+0000 7fb84ef79640 1 -- start start 2024-03-29T11:11:08.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.465+0000 7fb84ef79640 1 librados: setting wanted keys 2024-03-29T11:11:08.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.465+0000 7fb84ef79640 1 librados: calling monclient init 2024-03-29T11:11:08.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.465+0000 7fb84ef79640 10 monclient: init 2024-03-29T11:11:08.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.465+0000 7fb84ef79640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:08.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.465+0000 7fb84ef79640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:11:08.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.465+0000 7fb84ef79640 10 monclient(hunting): picked mon.noname-c con 0x7fb840006ec0 addr v1:172.21.15.71:6790/0 2024-03-29T11:11:08.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.465+0000 7fb84ef79640 10 monclient(hunting): picked mon.noname-a con 0x7fb8400cfcb0 addr v1:172.21.15.71:6789/0 2024-03-29T11:11:08.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.466+0000 7fb84d776640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fb8400cfcb0 0x7fb840164540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:33276/0 (socket says 172.21.15.71:33276) 2024-03-29T11:11:08.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.466+0000 7fb84ef79640 10 monclient(hunting): picked mon.noname-b con 0x7fb840164c50 addr v1:172.21.15.137:6789/0 2024-03-29T11:11:08.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.466+0000 7fb84d776640 1 -- 172.21.15.71:0/55977083 learned_addr learned my addr 172.21.15.71:0/55977083 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:11:08.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.466+0000 7fb84ef79640 1 -- 172.21.15.71:0/55977083 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8401640b0 con 0x7fb8400cfcb0 2024-03-29T11:11:08.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.466+0000 7fb84ef79640 1 -- 172.21.15.71:0/55977083 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb840031120 con 0x7fb840164c50 2024-03-29T11:11:08.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.466+0000 7fb84ef79640 1 -- 172.21.15.71:0/55977083 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb840031390 con 0x7fb840006ec0 2024-03-29T11:11:08.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.466+0000 7fb84ef79640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:08.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.466+0000 7fb84ef79640 10 monclient(hunting): authenticate will time out at 1424.725937s 2024-03-29T11:11:08.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.466+0000 7fb83e7fc640 10 client.?.objecter ms_handle_connect 0x7fb8400cfcb0 2024-03-29T11:11:08.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.466+0000 7fb83e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:08.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.466+0000 7fb83e7fc640 10 client.?.objecter ms_handle_connect 0x7fb840006ec0 2024-03-29T11:11:08.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.466+0000 7fb83e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:08.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.466+0000 7fb83e7fc640 1 -- 172.21.15.71:0/55977083 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1904991601 0 0) 0x7fb8401640b0 con 0x7fb8400cfcb0 2024-03-29T11:11:08.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.466+0000 7fb83e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:11:08.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.466+0000 7fb83e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:08.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.466+0000 7fb83e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:08.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.466+0000 7fb83e7fc640 10 monclient(hunting): my global_id is 4623 2024-03-29T11:11:08.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.467+0000 7fb83e7fc640 1 -- 172.21.15.71:0/55977083 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb82c004eb0 con 0x7fb8400cfcb0 2024-03-29T11:11:08.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.467+0000 7fb83e7fc640 1 -- 172.21.15.71:0/55977083 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017136490 0 0) 0x7fb840031390 con 0x7fb840006ec0 2024-03-29T11:11:08.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.467+0000 7fb83e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:11:08.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.467+0000 7fb83e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:08.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.467+0000 7fb83e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:08.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.467+0000 7fb83e7fc640 10 monclient(hunting): my global_id is 4691 2024-03-29T11:11:08.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.467+0000 7fb83e7fc640 1 -- 172.21.15.71:0/55977083 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8401640b0 con 0x7fb840006ec0 2024-03-29T11:11:08.461 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.467+0000 7fb83e7fc640 1 -- 172.21.15.71:0/55977083 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3318880212 0 0) 0x7fb82c004eb0 con 0x7fb8400cfcb0 2024-03-29T11:11:08.461 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.467+0000 7fb83e7fc640 1 -- 172.21.15.71:0/55977083 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb82c005490 con 0x7fb8400cfcb0 2024-03-29T11:11:08.461 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.467+0000 7fb83e7fc640 1 -- 172.21.15.71:0/55977083 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb84804fdd0 con 0x7fb8400cfcb0 2024-03-29T11:11:08.461 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.467+0000 7fb83e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:08.461 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.467+0000 7fb83e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:11:08.461 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.467+0000 7fb83e7fc640 10 monclient(hunting): dump: 2024-03-29T11:11:08.461 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:11:08.461 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:08.461 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:08.461 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:08.461 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:08.462 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:08.462 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:11:08.462 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:11:08.462 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:11:08.462 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:08.462 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.467+0000 7fb83e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:08.462 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.467+0000 7fb83e7fc640 1 -- 172.21.15.71:0/55977083 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1002675329 0 0) 0x7fb8401640b0 con 0x7fb840006ec0 2024-03-29T11:11:08.462 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.467+0000 7fb83e7fc640 1 -- 172.21.15.71:0/55977083 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb82c004eb0 con 0x7fb840006ec0 2024-03-29T11:11:08.462 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.467+0000 7fb83e7fc640 1 -- 172.21.15.71:0/55977083 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb844001b00 con 0x7fb840006ec0 2024-03-29T11:11:08.462 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.467+0000 7fb83e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:08.462 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.467+0000 7fb83e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:08.463 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.467+0000 7fb83e7fc640 10 monclient(hunting): dump: 2024-03-29T11:11:08.463 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:11:08.463 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:08.463 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:08.463 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:08.463 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:08.463 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:08.464 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:11:08.464 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:11:08.464 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:11:08.464 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:08.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.467+0000 7fb83e7fc640 1 -- 172.21.15.71:0/55977083 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2088335265 0 0) 0x7fb82c005490 con 0x7fb8400cfcb0 2024-03-29T11:11:08.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.467+0000 7fb83e7fc640 1 -- 172.21.15.71:0/55977083 >> v1:172.21.15.71:6790/0 conn(0x7fb840006ec0 legacy=0x7fb84002ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:08.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.467+0000 7fb83e7fc640 1 -- 172.21.15.71:0/55977083 >> v1:172.21.15.137:6789/0 conn(0x7fb840164c50 legacy=0x7fb84016b520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:11:08.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.468+0000 7fb83e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:11:08.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.468+0000 7fb83e7fc640 1 monclient: found mon.a 2024-03-29T11:11:08.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.468+0000 7fb83e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:08.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.468+0000 7fb83e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:11:08.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.468+0000 7fb83e7fc640 1 -- 172.21.15.71:0/55977083 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8400317b0 con 0x7fb8400cfcb0 2024-03-29T11:11:08.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.468+0000 7fb83e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:11:08.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.468+0000 7fb83e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:11:08.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.468+0000 7fb83e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:08.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.468+0000 7fb83e7fc640 1 -- 172.21.15.71:0/55977083 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb84804f400 con 0x7fb8400cfcb0 2024-03-29T11:11:08.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.468+0000 7fb83e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:08.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.468+0000 7fb83e7fc640 1 -- 172.21.15.71:0/55977083 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fb848075710 con 0x7fb8400cfcb0 2024-03-29T11:11:08.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.468+0000 7fb83e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:08.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.468+0000 7fb83e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:11:08.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.468+0000 7fb83e7fc640 10 monclient: dump: 2024-03-29T11:11:08.466 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:11:08.466 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:08.466 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:08.466 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:08.466 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:08.466 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:08.466 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:11:08.466 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:11:08.467 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:11:08.467 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:08.467 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.468+0000 7fb83ffff640 4 set_mon_vals no callback set 2024-03-29T11:11:08.467 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.469+0000 7fb84ef79640 5 monclient: authenticate success, global_id 4623 2024-03-29T11:11:08.467 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.469+0000 7fb84ef79640 10 monclient: _renew_subs 2024-03-29T11:11:08.467 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.469+0000 7fb84ef79640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:11:08.467 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.469+0000 7fb84ef79640 1 -- 172.21.15.71:0/55977083 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8400319a0 con 0x7fb8400cfcb0 2024-03-29T11:11:08.467 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.469+0000 7fb84ef79640 10 client.4623.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:08.467 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.469+0000 7fb84ef79640 10 monclient: _renew_subs 2024-03-29T11:11:08.467 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.469+0000 7fb84ef79640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:11:08.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.469+0000 7fb84ef79640 1 -- 172.21.15.71:0/55977083 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb84002f8f0 con 0x7fb8400cfcb0 2024-03-29T11:11:08.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.469+0000 7fb84ef79640 1 librados: init done 2024-03-29T11:11:08.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.470+0000 7fb84ef79640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:08.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.471+0000 7fb84ef79640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:08.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.471+0000 7fb84ef79640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:11:08.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.471+0000 7fb84ef79640 1 -- 172.21.15.71:0/55977083 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb840164260 con 0x7fb8400cfcb0 2024-03-29T11:11:08.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.472+0000 7fb83e7fc640 1 -- 172.21.15.71:0/55977083 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7fb848067bf0 con 0x7fb8400cfcb0 2024-03-29T11:11:08.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.473+0000 7fb83e7fc640 1 -- 172.21.15.71:0/55977083 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 45029510 0 0) 0x7fb8480ff3b0 con 0x7fb8400cfcb0 2024-03-29T11:11:08.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.473+0000 7fb83e7fc640 10 client.4623.objecter ms_dispatch 0x7fb848000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:11:08.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.473+0000 7fb83e7fc640 3 client.4623.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:11:08.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.473+0000 7fb83e7fc640 3 client.4623.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:11:08.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.473+0000 7fb83e7fc640 20 client.4623.objecter dump_active .. 0 homeless 2024-03-29T11:11:08.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.473+0000 7fb84df77640 10 monclient: get_auth_request con 0x7fb82c06e0c0 auth_method 0 2024-03-29T11:11:08.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.474+0000 7fb83e7fc640 10 client.4623.objecter ms_handle_connect 0x7fb82c06e0c0 2024-03-29T11:11:08.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.479+0000 7fb83e7fc640 1 -- 172.21.15.71:0/55977083 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb8480cacf0 con 0x7fb8400cfcb0 2024-03-29T11:11:08.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.479+0000 7fb83e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:08.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.479+0000 7fb83e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:08.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.794+0000 7fb84ef79640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:08.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.794+0000 7fb84ef79640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:08.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.794+0000 7fb84ef79640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:11:08.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.794+0000 7fb84ef79640 1 -- 172.21.15.71:0/55977083 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb840164260 con 0x7fb8400cfcb0 2024-03-29T11:11:08.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.796+0000 7fb83e7fc640 1 -- 172.21.15.71:0/55977083 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 577002532) 0x7fb840164260 con 0x7fb8400cfcb0 2024-03-29T11:11:08.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.796+0000 7fb83e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:08.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.796+0000 7fb83e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:08.776 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:11:08.776 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:10:37.063243+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:10:37.063243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:08.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.800+0000 7fb84ef79640 10 librados: watch_flush enter 2024-03-29T11:11:08.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.800+0000 7fb84ef79640 10 librados: watch_flush exit 2024-03-29T11:11:08.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.800+0000 7fb84ef79640 20 client.4623.objecter shutdown clearing up homeless session... 2024-03-29T11:11:08.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.800+0000 7fb84ef79640 10 client.4623.objecter successfully canceled tick 2024-03-29T11:11:08.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.801+0000 7fb84ef79640 1 -- 172.21.15.71:0/55977083 >> v1:172.21.15.137:6816/973332665 conn(0x7fb82c06e0c0 legacy=0x7fb82c070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:08.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.801+0000 7fb84ef79640 10 monclient: shutdown 2024-03-29T11:11:08.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.801+0000 7fb84ef79640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:08.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.801+0000 7fb84ef79640 1 -- 172.21.15.71:0/55977083 >> v1:172.21.15.71:6789/0 conn(0x7fb8400cfcb0 legacy=0x7fb840164540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:08.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.801+0000 7fb84ef79640 1 -- 172.21.15.71:0/55977083 shutdown_connections 2024-03-29T11:11:08.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.801+0000 7fb84ef79640 1 -- 172.21.15.71:0/55977083 >> 172.21.15.71:0/55977083 conn(0x7fb840005d40 msgr2=0x7fb8400378e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:08.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.801+0000 7fb84ef79640 1 -- 172.21.15.71:0/55977083 shutdown_connections 2024-03-29T11:11:08.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.802+0000 7fb84ef79640 1 -- 172.21.15.71:0/55977083 wait complete. 2024-03-29T11:11:08.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:08.802+0000 7fb84ef79640 1 librados: shutdown 2024-03-29T11:11:08.799 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:11:15.606 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:15.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.773+0000 7f4b7f0c1640 10 monclient: get_monmap_and_config 2024-03-29T11:11:15.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.773+0000 7f4b7f0c1640 10 monclient: build_initial_monmap 2024-03-29T11:11:15.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.773+0000 7f4b7f0c1640 10 monclient: monmap: 2024-03-29T11:11:15.752 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:11:15.752 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:15.752 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:11:15.774673+0000 2024-03-29T11:11:15.752 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:11:15.774673+0000 2024-03-29T11:11:15.752 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:15.752 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:15.752 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:11:15.752 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:11:15.752 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:11:15.752 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:15.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.774+0000 7f4b7f0c1640 1 Processor -- start 2024-03-29T11:11:15.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.774+0000 7f4b7f0c1640 1 -- start start 2024-03-29T11:11:15.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.774+0000 7f4b7f0c1640 10 monclient: init 2024-03-29T11:11:15.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.775+0000 7f4b7f0c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:15.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.775+0000 7f4b7f0c1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:11:15.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.776+0000 7f4b7ce5d640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f4b78074650 0x7f4b781498e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:55170/0 (socket says 172.21.15.71:55170) 2024-03-29T11:11:15.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.776+0000 7f4b7ce5d640 1 -- 172.21.15.71:0/3972651703 learned_addr learned my addr 172.21.15.71:0/3972651703 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:11:15.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.776+0000 7f4b7f0c1640 10 monclient(hunting): picked mon.noname-b con 0x7f4b78074650 addr v1:172.21.15.71:6790/0 2024-03-29T11:11:15.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.776+0000 7f4b7f0c1640 10 monclient(hunting): picked mon.noname-c con 0x7f4b78149ff0 addr v1:172.21.15.137:6789/0 2024-03-29T11:11:15.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.776+0000 7f4b7f0c1640 10 monclient(hunting): picked mon.noname-a con 0x7f4b78072620 addr v1:172.21.15.71:6789/0 2024-03-29T11:11:15.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.776+0000 7f4b7f0c1640 1 -- 172.21.15.71:0/3972651703 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b7806ecf0 con 0x7f4b78072620 2024-03-29T11:11:15.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.776+0000 7f4b7f0c1640 1 -- 172.21.15.71:0/3972651703 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b78070440 con 0x7f4b78149ff0 2024-03-29T11:11:15.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.776+0000 7f4b7f0c1640 1 -- 172.21.15.71:0/3972651703 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b78070720 con 0x7f4b78074650 2024-03-29T11:11:15.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.776+0000 7f4b7f0c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:15.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.776+0000 7f4b7f0c1640 10 monclient(hunting): authenticate will time out at 1432.034516s 2024-03-29T11:11:15.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.776+0000 7f4b777fe640 1 -- 172.21.15.71:0/3972651703 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322324890 0 0) 0x7f4b78070720 con 0x7f4b78074650 2024-03-29T11:11:15.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.776+0000 7f4b777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:11:15.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.776+0000 7f4b777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:15.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.776+0000 7f4b777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:15.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.776+0000 7f4b777fe640 10 monclient(hunting): my global_id is 4694 2024-03-29T11:11:15.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 1 -- 172.21.15.71:0/3972651703 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b60003520 con 0x7f4b78074650 2024-03-29T11:11:15.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 1 -- 172.21.15.71:0/3972651703 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2109989793 0 0) 0x7f4b78070440 con 0x7f4b78149ff0 2024-03-29T11:11:15.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:11:15.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:15.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:15.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:11:15.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 1 -- 172.21.15.71:0/3972651703 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b78070720 con 0x7f4b78149ff0 2024-03-29T11:11:15.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 1 -- 172.21.15.71:0/3972651703 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 506044219 0 0) 0x7f4b7806ecf0 con 0x7f4b78072620 2024-03-29T11:11:15.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:11:15.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:15.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:15.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 10 monclient(hunting): my global_id is 4626 2024-03-29T11:11:15.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 1 -- 172.21.15.71:0/3972651703 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b78070440 con 0x7f4b78072620 2024-03-29T11:11:15.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 1 -- 172.21.15.71:0/3972651703 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 246889067 0 0) 0x7f4b60003520 con 0x7f4b78074650 2024-03-29T11:11:15.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 1 -- 172.21.15.71:0/3972651703 >> v1:172.21.15.137:6789/0 conn(0x7f4b78149ff0 legacy=0x7f4b78071f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:15.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 1 -- 172.21.15.71:0/3972651703 >> v1:172.21.15.71:6789/0 conn(0x7f4b78072620 legacy=0x7f4b78151c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:15.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 10 monclient: _finish_hunting 0 2024-03-29T11:11:15.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 1 monclient: found mon.noname-b 2024-03-29T11:11:15.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:15.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:11:15.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 1 -- 172.21.15.71:0/3972651703 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7806e730 con 0x7f4b78074650 2024-03-29T11:11:15.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 10 monclient: _finish_auth 0 2024-03-29T11:11:15.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 10 monclient: _check_auth_tickets 2024-03-29T11:11:15.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:15.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 1 -- 172.21.15.71:0/3972651703 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4b70001990 con 0x7f4b78074650 2024-03-29T11:11:15.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:15.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:11:15.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.777+0000 7f4b777fe640 10 monclient: dump: 2024-03-29T11:11:15.771 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:11:15.771 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:15.771 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:15.771 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:15.771 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:15.771 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:15.772 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:11:15.772 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:11:15.772 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:11:15.772 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:15.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.778+0000 7f4b7f0c1640 5 monclient: authenticate success, global_id 4694 2024-03-29T11:11:15.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.778+0000 7f4b7f0c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:15.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.778+0000 7f4b777fe640 1 -- 172.21.15.71:0/3972651703 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b70001bb0 con 0x7f4b78074650 2024-03-29T11:11:15.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.778+0000 7f4b777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:15.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.778+0000 7f4b777fe640 1 -- 172.21.15.71:0/3972651703 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4b70001f40 con 0x7f4b78074650 2024-03-29T11:11:15.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.778+0000 7f4b777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:15.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.778+0000 7f4b777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:15.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.778+0000 7f4b777fe640 10 monclient: dump: 2024-03-29T11:11:15.772 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:11:15.772 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:15.773 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:15.773 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:15.773 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:15.773 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:15.773 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:11:15.773 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:11:15.773 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:11:15.773 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:15.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.778+0000 7f4b7f0c1640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:15.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.778+0000 7f4b7f0c1640 4 set_mon_vals no callback set 2024-03-29T11:11:15.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.779+0000 7f4b7f0c1640 10 monclient: shutdown 2024-03-29T11:11:15.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.779+0000 7f4b7f0c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:15.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.779+0000 7f4b7f0c1640 1 -- 172.21.15.71:0/3972651703 >> v1:172.21.15.71:6790/0 conn(0x7f4b78074650 legacy=0x7f4b781498e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:15.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.779+0000 7f4b7f0c1640 1 -- 172.21.15.71:0/3972651703 shutdown_connections 2024-03-29T11:11:15.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.779+0000 7f4b7f0c1640 1 -- 172.21.15.71:0/3972651703 >> 172.21.15.71:0/3972651703 conn(0x7f4b780b3f40 msgr2=0x7f4b780b4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:15.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.779+0000 7f4b7f0c1640 1 -- 172.21.15.71:0/3972651703 shutdown_connections 2024-03-29T11:11:15.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.779+0000 7f4b7f0c1640 1 -- 172.21.15.71:0/3972651703 wait complete. 2024-03-29T11:11:15.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.780+0000 7f4b7f0c1640 10 monclient: build_initial_monmap 2024-03-29T11:11:15.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.780+0000 7f4b7f0c1640 10 monclient: monmap: 2024-03-29T11:11:15.774 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:11:15.774 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:15.774 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:11:15.774 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:11:15.774 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:15.774 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:15.774 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:11:15.774 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:11:15.774 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:11:15.774 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:15.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.781+0000 7f4b7f0c1640 1 librados: starting msgr at 2024-03-29T11:11:15.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.781+0000 7f4b7f0c1640 1 librados: starting objecter 2024-03-29T11:11:15.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.781+0000 7f4b7f0c1640 1 Processor -- start 2024-03-29T11:11:15.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.781+0000 7f4b7f0c1640 1 -- start start 2024-03-29T11:11:15.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.781+0000 7f4b7f0c1640 1 librados: setting wanted keys 2024-03-29T11:11:15.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.781+0000 7f4b7f0c1640 1 librados: calling monclient init 2024-03-29T11:11:15.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.781+0000 7f4b7f0c1640 10 monclient: init 2024-03-29T11:11:15.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.782+0000 7f4b7f0c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:15.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.782+0000 7f4b7f0c1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:11:15.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.782+0000 7f4b7f0c1640 10 monclient(hunting): picked mon.noname-b con 0x7f4b78072620 addr v1:172.21.15.137:6789/0 2024-03-29T11:11:15.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.782+0000 7f4b7f0c1640 10 monclient(hunting): picked mon.noname-a con 0x7f4b78074650 addr v1:172.21.15.71:6789/0 2024-03-29T11:11:15.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.782+0000 7f4b7f0c1640 10 monclient(hunting): picked mon.noname-c con 0x7f4b780ae500 addr v1:172.21.15.71:6790/0 2024-03-29T11:11:15.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.782+0000 7f4b7ce5d640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f4b78072620 0x7f4b781448d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.71:33722/0 (socket says 172.21.15.71:33722) 2024-03-29T11:11:15.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.782+0000 7f4b7ce5d640 1 -- 172.21.15.71:0/1656592248 learned_addr learned my addr 172.21.15.71:0/1656592248 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:11:15.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.783+0000 7f4b7f0c1640 1 -- 172.21.15.71:0/1656592248 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b7813e660 con 0x7f4b78074650 2024-03-29T11:11:15.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.783+0000 7f4b7f0c1640 1 -- 172.21.15.71:0/1656592248 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b78144160 con 0x7f4b78072620 2024-03-29T11:11:15.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.783+0000 7f4b757fa640 10 client.?.objecter ms_handle_connect 0x7f4b780ae500 2024-03-29T11:11:15.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.783+0000 7f4b757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:15.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.783+0000 7f4b7f0c1640 1 -- 172.21.15.71:0/1656592248 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b78146480 con 0x7f4b780ae500 2024-03-29T11:11:15.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.783+0000 7f4b7f0c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:15.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.783+0000 7f4b7f0c1640 10 monclient(hunting): authenticate will time out at 1432.041389s 2024-03-29T11:11:15.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.783+0000 7f4b757fa640 10 client.?.objecter ms_handle_connect 0x7f4b78072620 2024-03-29T11:11:15.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.783+0000 7f4b757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:15.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.783+0000 7f4b757fa640 10 client.?.objecter ms_handle_connect 0x7f4b78074650 2024-03-29T11:11:15.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.783+0000 7f4b757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:15.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.784+0000 7f4b757fa640 1 -- 172.21.15.71:0/1656592248 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297663637 0 0) 0x7f4b78144160 con 0x7f4b78072620 2024-03-29T11:11:15.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.784+0000 7f4b757fa640 10 monclient(hunting): _negotiate 2024-03-29T11:11:15.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.784+0000 7f4b757fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:15.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.784+0000 7f4b757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:15.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.784+0000 7f4b757fa640 10 monclient(hunting): my global_id is 4543 2024-03-29T11:11:15.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.784+0000 7f4b757fa640 1 -- 172.21.15.71:0/1656592248 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b64004eb0 con 0x7f4b78072620 2024-03-29T11:11:15.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.784+0000 7f4b757fa640 1 -- 172.21.15.71:0/1656592248 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325176500 0 0) 0x7f4b78070720 con 0x7f4b78074650 2024-03-29T11:11:15.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.784+0000 7f4b757fa640 10 monclient(hunting): _negotiate 2024-03-29T11:11:15.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.784+0000 7f4b757fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:15.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.784+0000 7f4b757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:15.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.784+0000 7f4b757fa640 10 monclient(hunting): my global_id is 4629 2024-03-29T11:11:15.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.784+0000 7f4b757fa640 1 -- 172.21.15.71:0/1656592248 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b78144160 con 0x7f4b78074650 2024-03-29T11:11:15.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.784+0000 7f4b757fa640 1 -- 172.21.15.71:0/1656592248 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74919279 0 0) 0x7f4b78146480 con 0x7f4b780ae500 2024-03-29T11:11:15.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.784+0000 7f4b757fa640 10 monclient(hunting): _negotiate 2024-03-29T11:11:15.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.784+0000 7f4b757fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:15.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.784+0000 7f4b757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:15.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.784+0000 7f4b757fa640 10 monclient(hunting): my global_id is 4697 2024-03-29T11:11:15.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.784+0000 7f4b757fa640 1 -- 172.21.15.71:0/1656592248 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b78070720 con 0x7f4b780ae500 2024-03-29T11:11:15.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.784+0000 7f4b757fa640 1 -- 172.21.15.71:0/1656592248 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3372038295 0 0) 0x7f4b64004eb0 con 0x7f4b78072620 2024-03-29T11:11:15.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b757fa640 1 -- 172.21.15.71:0/1656592248 --> v1:172.21.15.137:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b78146480 con 0x7f4b78072620 2024-03-29T11:11:15.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b757fa640 1 -- 172.21.15.71:0/1656592248 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3070272231 0 0) 0x7f4b78144160 con 0x7f4b78074650 2024-03-29T11:11:15.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b757fa640 1 -- 172.21.15.71:0/1656592248 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b64004eb0 con 0x7f4b78074650 2024-03-29T11:11:15.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b757fa640 1 -- 172.21.15.71:0/1656592248 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4b70001990 con 0x7f4b78072620 2024-03-29T11:11:15.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:15.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b757fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:11:15.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b757fa640 10 monclient(hunting): dump: 2024-03-29T11:11:15.778 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:11:15.778 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:15.778 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:15.778 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:15.778 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:15.778 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:15.778 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:11:15.778 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:11:15.779 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:11:15.779 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:15.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b757fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:15.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b757fa640 1 -- 172.21.15.71:0/1656592248 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4b6c0016f0 con 0x7f4b78074650 2024-03-29T11:11:15.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:15.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b757fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:11:15.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b757fa640 10 monclient(hunting): dump: 2024-03-29T11:11:15.779 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:11:15.779 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:15.779 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:15.779 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:15.779 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:15.779 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:15.779 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:11:15.779 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:11:15.780 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:11:15.780 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:15.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b757fa640 1 -- 172.21.15.71:0/1656592248 <== mon.1 v1:172.21.15.137:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1911602719 0 0) 0x7f4b78146480 con 0x7f4b78072620 2024-03-29T11:11:15.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b757fa640 1 -- 172.21.15.71:0/1656592248 >> v1:172.21.15.71:6790/0 conn(0x7f4b780ae500 legacy=0x7f4b781e6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:15.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b757fa640 1 -- 172.21.15.71:0/1656592248 >> v1:172.21.15.71:6789/0 conn(0x7f4b78074650 legacy=0x7f4b78148eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:15.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b757fa640 10 monclient: _finish_hunting 0 2024-03-29T11:11:15.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b757fa640 1 monclient: found mon.b 2024-03-29T11:11:15.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:15.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b757fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:11:15.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b757fa640 1 -- 172.21.15.71:0/1656592248 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b781443b0 con 0x7f4b78072620 2024-03-29T11:11:15.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b757fa640 10 monclient: _finish_auth 0 2024-03-29T11:11:15.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b757fa640 10 monclient: _check_auth_tickets 2024-03-29T11:11:15.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:15.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b7f0c1640 5 monclient: authenticate success, global_id 4543 2024-03-29T11:11:15.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b7f0c1640 10 monclient: _renew_subs 2024-03-29T11:11:15.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b7f0c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:11:15.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b7f0c1640 1 -- 172.21.15.71:0/1656592248 --> v1:172.21.15.137:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b78144fe0 con 0x7f4b78072620 2024-03-29T11:11:15.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b7f0c1640 10 client.4543.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:15.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b7f0c1640 10 monclient: _renew_subs 2024-03-29T11:11:15.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b7f0c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:11:15.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b7f0c1640 1 -- 172.21.15.71:0/1656592248 --> v1:172.21.15.137:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b781497a0 con 0x7f4b78072620 2024-03-29T11:11:15.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.785+0000 7f4b7f0c1640 1 librados: init done 2024-03-29T11:11:15.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.786+0000 7f4b757fa640 1 -- 172.21.15.71:0/1656592248 <== mon.1 v1:172.21.15.137:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b70004e80 con 0x7f4b78072620 2024-03-29T11:11:15.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.786+0000 7f4b757fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:15.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.786+0000 7f4b76ffd640 4 set_mon_vals no callback set 2024-03-29T11:11:15.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.788+0000 7f4b757fa640 1 -- 172.21.15.71:0/1656592248 <== mon.1 v1:172.21.15.137:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f4b70005290 con 0x7f4b78072620 2024-03-29T11:11:15.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.788+0000 7f4b757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:15.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.788+0000 7f4b757fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:15.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.788+0000 7f4b757fa640 10 monclient: dump: 2024-03-29T11:11:15.782 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:11:15.782 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:15.782 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:15.782 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:15.782 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:15.782 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:15.782 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:11:15.782 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:11:15.782 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:11:15.782 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:15.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.789+0000 7f4b757fa640 1 -- 172.21.15.71:0/1656592248 <== mon.1 v1:172.21.15.137:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f4b700054b0 con 0x7f4b78072620 2024-03-29T11:11:15.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.789+0000 7f4b7f0c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:15.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.789+0000 7f4b7f0c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:15.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.789+0000 7f4b7f0c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:11:15.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.789+0000 7f4b7f0c1640 1 -- 172.21.15.71:0/1656592248 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b58005080 con 0x7f4b78072620 2024-03-29T11:11:15.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.789+0000 7f4b757fa640 1 -- 172.21.15.71:0/1656592248 <== mon.1 v1:172.21.15.137:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 45029510 0 0) 0x7f4b70084f40 con 0x7f4b78072620 2024-03-29T11:11:15.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.789+0000 7f4b757fa640 10 client.4543.objecter ms_dispatch 0x7f4b78000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:11:15.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.789+0000 7f4b757fa640 3 client.4543.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:11:15.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.789+0000 7f4b757fa640 3 client.4543.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:11:15.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.789+0000 7f4b757fa640 20 client.4543.objecter dump_active .. 0 homeless 2024-03-29T11:11:15.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.792+0000 7f4b77fff640 10 monclient: get_auth_request con 0x7f4b6406dfb0 auth_method 0 2024-03-29T11:11:15.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.792+0000 7f4b757fa640 10 client.4543.objecter ms_handle_connect 0x7f4b6406dfb0 2024-03-29T11:11:15.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.799+0000 7f4b757fa640 1 -- 172.21.15.71:0/1656592248 <== mon.1 v1:172.21.15.137:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b700508e0 con 0x7f4b78072620 2024-03-29T11:11:15.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.799+0000 7f4b757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:15.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:15.799+0000 7f4b757fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:16.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:16.178+0000 7f4b7f0c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:16.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:16.178+0000 7f4b7f0c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:16.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:16.178+0000 7f4b7f0c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.137:6789/0 2024-03-29T11:11:16.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:16.178+0000 7f4b7f0c1640 1 -- 172.21.15.71:0/1656592248 --> v1:172.21.15.137:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b58005330 con 0x7f4b78072620 2024-03-29T11:11:16.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:16.179+0000 7f4b757fa640 1 -- 172.21.15.71:0/1656592248 <== mon.1 v1:172.21.15.137:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 577002532) 0x7f4b70052810 con 0x7f4b78072620 2024-03-29T11:11:16.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:16.179+0000 7f4b757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:16.158 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:11:16.158 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:10:37.063243+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:10:37.063243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:16.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:16.179+0000 7f4b757fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:16.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:16.183+0000 7f4b7f0c1640 10 librados: watch_flush enter 2024-03-29T11:11:16.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:16.183+0000 7f4b7f0c1640 10 librados: watch_flush exit 2024-03-29T11:11:16.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:16.183+0000 7f4b7f0c1640 20 client.4543.objecter shutdown clearing up homeless session... 2024-03-29T11:11:16.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:16.183+0000 7f4b7f0c1640 10 client.4543.objecter successfully canceled tick 2024-03-29T11:11:16.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:16.183+0000 7f4b7f0c1640 1 -- 172.21.15.71:0/1656592248 >> v1:172.21.15.137:6816/973332665 conn(0x7f4b6406dfb0 legacy=0x7f4b64070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:16.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:16.183+0000 7f4b7f0c1640 10 monclient: shutdown 2024-03-29T11:11:16.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:16.183+0000 7f4b7f0c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:16.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:16.183+0000 7f4b7f0c1640 1 -- 172.21.15.71:0/1656592248 >> v1:172.21.15.137:6789/0 conn(0x7f4b78072620 legacy=0x7f4b781448d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:16.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:16.183+0000 7f4b7f0c1640 1 -- 172.21.15.71:0/1656592248 shutdown_connections 2024-03-29T11:11:16.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:16.183+0000 7f4b7f0c1640 1 -- 172.21.15.71:0/1656592248 >> 172.21.15.71:0/1656592248 conn(0x7f4b780b3f40 msgr2=0x7f4b780b4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:16.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:16.184+0000 7f4b7f0c1640 1 -- 172.21.15.71:0/1656592248 shutdown_connections 2024-03-29T11:11:16.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:16.184+0000 7f4b7f0c1640 1 -- 172.21.15.71:0/1656592248 wait complete. 2024-03-29T11:11:16.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:16.184+0000 7f4b7f0c1640 1 librados: shutdown 2024-03-29T11:11:16.176 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:11:22.983 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:23.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.154+0000 7fa4310d0640 10 monclient: get_monmap_and_config 2024-03-29T11:11:23.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.154+0000 7fa4310d0640 10 monclient: build_initial_monmap 2024-03-29T11:11:23.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.154+0000 7fa4310d0640 10 monclient: monmap: 2024-03-29T11:11:23.133 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:11:23.133 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:23.133 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:11:23.155300+0000 2024-03-29T11:11:23.133 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:11:23.155300+0000 2024-03-29T11:11:23.133 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:23.133 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:23.133 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:11:23.133 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:11:23.133 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:11:23.133 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:23.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.156+0000 7fa4310d0640 1 Processor -- start 2024-03-29T11:11:23.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.157+0000 7fa4310d0640 1 -- start start 2024-03-29T11:11:23.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.157+0000 7fa4310d0640 10 monclient: init 2024-03-29T11:11:23.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.157+0000 7fa4310d0640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:23.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.157+0000 7fa4310d0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:11:23.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.158+0000 7fa4310d0640 10 monclient(hunting): picked mon.noname-a con 0x7fa42c0b0770 addr v1:172.21.15.71:6789/0 2024-03-29T11:11:23.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.158+0000 7fa4310d0640 10 monclient(hunting): picked mon.noname-c con 0x7fa42c14a700 addr v1:172.21.15.137:6789/0 2024-03-29T11:11:23.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.158+0000 7fa4310d0640 10 monclient(hunting): picked mon.noname-b con 0x7fa42c0ac5a0 addr v1:172.21.15.71:6790/0 2024-03-29T11:11:23.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.158+0000 7fa4310d0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa42c0aba70 con 0x7fa42c0b0770 2024-03-29T11:11:23.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.158+0000 7fa4310d0640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa42c0aecb0 con 0x7fa42c14a700 2024-03-29T11:11:23.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.158+0000 7fa4310d0640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa42c0af150 con 0x7fa42c0ac5a0 2024-03-29T11:11:23.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.158+0000 7fa4310d0640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:23.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.158+0000 7fa4310d0640 10 monclient(hunting): authenticate will time out at 1439.416177s 2024-03-29T11:11:23.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.158+0000 7fa42b59e640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fa42c0ac5a0 0x7fa42c151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:35294/0 (socket says 172.21.15.71:35294) 2024-03-29T11:11:23.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.158+0000 7fa42b59e640 1 -- 172.21.15.71:0/375454986 learned_addr learned my addr 172.21.15.71:0/375454986 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:11:23.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.158+0000 7fa42a59c640 1 -- 172.21.15.71:0/375454986 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2099615736 0 0) 0x7fa42c0af150 con 0x7fa42c0ac5a0 2024-03-29T11:11:23.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.159+0000 7fa42a59c640 10 monclient(hunting): _negotiate 2024-03-29T11:11:23.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.159+0000 7fa42a59c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:23.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.159+0000 7fa42a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:23.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.159+0000 7fa42a59c640 10 monclient(hunting): my global_id is 4700 2024-03-29T11:11:23.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.159+0000 7fa42a59c640 1 -- 172.21.15.71:0/375454986 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa418003520 con 0x7fa42c0ac5a0 2024-03-29T11:11:23.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.160+0000 7fa42a59c640 1 -- 172.21.15.71:0/375454986 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1906471413 0 0) 0x7fa418003520 con 0x7fa42c0ac5a0 2024-03-29T11:11:23.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.160+0000 7fa42a59c640 1 -- 172.21.15.71:0/375454986 >> v1:172.21.15.137:6789/0 conn(0x7fa42c14a700 legacy=0x7fa42c0abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:11:23.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.162+0000 7fa42a59c640 1 -- 172.21.15.71:0/375454986 >> v1:172.21.15.71:6789/0 conn(0x7fa42c0b0770 legacy=0x7fa42c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:23.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.162+0000 7fa42a59c640 10 monclient: _finish_hunting 0 2024-03-29T11:11:23.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.162+0000 7fa42a59c640 1 monclient: found mon.noname-b 2024-03-29T11:11:23.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.162+0000 7fa42a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:23.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.162+0000 7fa42a59c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:11:23.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.162+0000 7fa42a59c640 1 -- 172.21.15.71:0/375454986 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa42c0ac9d0 con 0x7fa42c0ac5a0 2024-03-29T11:11:23.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.162+0000 7fa42a59c640 10 monclient: _finish_auth 0 2024-03-29T11:11:23.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.162+0000 7fa42a59c640 10 monclient: _check_auth_tickets 2024-03-29T11:11:23.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.162+0000 7fa42a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:23.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.163+0000 7fa42a59c640 1 -- 172.21.15.71:0/375454986 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fa424001a10 con 0x7fa42c0ac5a0 2024-03-29T11:11:23.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.163+0000 7fa42a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:23.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.163+0000 7fa42a59c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:11:23.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.163+0000 7fa42a59c640 10 monclient: dump: 2024-03-29T11:11:23.143 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:11:23.143 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:23.144 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:23.144 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:23.144 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:23.144 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:23.144 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:11:23.144 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:11:23.144 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:11:23.144 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:23.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.163+0000 7fa42a59c640 1 -- 172.21.15.71:0/375454986 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa424001ca0 con 0x7fa42c0ac5a0 2024-03-29T11:11:23.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.163+0000 7fa42a59c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:23.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.163+0000 7fa42a59c640 1 -- 172.21.15.71:0/375454986 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fa424002030 con 0x7fa42c0ac5a0 2024-03-29T11:11:23.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.163+0000 7fa42a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:23.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.164+0000 7fa42a59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:23.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.164+0000 7fa42a59c640 10 monclient: dump: 2024-03-29T11:11:23.145 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:11:23.145 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:23.145 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:23.145 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:23.145 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:23.145 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:23.145 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:11:23.145 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:11:23.145 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:11:23.145 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:23.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.164+0000 7fa4310d0640 5 monclient: authenticate success, global_id 4700 2024-03-29T11:11:23.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.164+0000 7fa4310d0640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:23.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.164+0000 7fa4310d0640 4 set_mon_vals no callback set 2024-03-29T11:11:23.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.165+0000 7fa4310d0640 10 monclient: shutdown 2024-03-29T11:11:23.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.165+0000 7fa4310d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:23.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.165+0000 7fa4310d0640 1 -- 172.21.15.71:0/375454986 >> v1:172.21.15.71:6790/0 conn(0x7fa42c0ac5a0 legacy=0x7fa42c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:23.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.165+0000 7fa4310d0640 1 -- 172.21.15.71:0/375454986 shutdown_connections 2024-03-29T11:11:23.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.165+0000 7fa4310d0640 1 -- 172.21.15.71:0/375454986 >> 172.21.15.71:0/375454986 conn(0x7fa42c06b660 msgr2=0x7fa42c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:23.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.166+0000 7fa4310d0640 1 -- 172.21.15.71:0/375454986 shutdown_connections 2024-03-29T11:11:23.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.168+0000 7fa4310d0640 1 -- 172.21.15.71:0/375454986 wait complete. 2024-03-29T11:11:23.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.169+0000 7fa4310d0640 10 monclient: build_initial_monmap 2024-03-29T11:11:23.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.169+0000 7fa4310d0640 10 monclient: monmap: 2024-03-29T11:11:23.148 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:11:23.148 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:23.148 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:11:23.148 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:11:23.148 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:23.148 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:23.148 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:11:23.148 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:11:23.148 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:11:23.148 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:23.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.169+0000 7fa4310d0640 1 librados: starting msgr at 2024-03-29T11:11:23.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.169+0000 7fa4310d0640 1 librados: starting objecter 2024-03-29T11:11:23.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.170+0000 7fa4310d0640 1 Processor -- start 2024-03-29T11:11:23.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.170+0000 7fa4310d0640 1 -- start start 2024-03-29T11:11:23.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.170+0000 7fa4310d0640 1 librados: setting wanted keys 2024-03-29T11:11:23.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.170+0000 7fa4310d0640 1 librados: calling monclient init 2024-03-29T11:11:23.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.170+0000 7fa4310d0640 10 monclient: init 2024-03-29T11:11:23.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.170+0000 7fa4310d0640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:23.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.170+0000 7fa4310d0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:11:23.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.170+0000 7fa4310d0640 10 monclient(hunting): picked mon.noname-b con 0x7fa42c0ac5a0 addr v1:172.21.15.137:6789/0 2024-03-29T11:11:23.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.170+0000 7fa4310d0640 10 monclient(hunting): picked mon.noname-a con 0x7fa42c0b0770 addr v1:172.21.15.71:6789/0 2024-03-29T11:11:23.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.170+0000 7fa4310d0640 10 monclient(hunting): picked mon.noname-c con 0x7fa42c14a700 addr v1:172.21.15.71:6790/0 2024-03-29T11:11:23.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.170+0000 7fa4310d0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa42c1239d0 con 0x7fa42c0b0770 2024-03-29T11:11:23.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.170+0000 7fa4310d0640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa42c125c40 con 0x7fa42c0ac5a0 2024-03-29T11:11:23.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.170+0000 7fa4310d0640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa42c145a50 con 0x7fa42c14a700 2024-03-29T11:11:23.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.170+0000 7fa4310d0640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:23.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.170+0000 7fa4310d0640 10 monclient(hunting): authenticate will time out at 1439.429095s 2024-03-29T11:11:23.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.173+0000 7fa42b59e640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fa42c14a700 0x7fa42c1db700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:36686/0 (socket says 172.21.15.71:36686) 2024-03-29T11:11:23.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.173+0000 7fa42b59e640 1 -- 172.21.15.71:0/2336005193 learned_addr learned my addr 172.21.15.71:0/2336005193 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:11:23.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.173+0000 7fa4237fe640 10 client.?.objecter ms_handle_connect 0x7fa42c14a700 2024-03-29T11:11:23.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.174+0000 7fa4237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:23.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.174+0000 7fa4237fe640 10 client.?.objecter ms_handle_connect 0x7fa42c0b0770 2024-03-29T11:11:23.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.174+0000 7fa4237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:23.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.174+0000 7fa4237fe640 1 -- 172.21.15.71:0/2336005193 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212247659 0 0) 0x7fa42c145a50 con 0x7fa42c14a700 2024-03-29T11:11:23.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.174+0000 7fa4237fe640 10 monclient(hunting): _negotiate 2024-03-29T11:11:23.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.174+0000 7fa4237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:23.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.174+0000 7fa4237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:23.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.174+0000 7fa4237fe640 10 monclient(hunting): my global_id is 4703 2024-03-29T11:11:23.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.174+0000 7fa4237fe640 1 -- 172.21.15.71:0/2336005193 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa40c004eb0 con 0x7fa42c14a700 2024-03-29T11:11:23.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.174+0000 7fa4237fe640 1 -- 172.21.15.71:0/2336005193 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3160378123 0 0) 0x7fa42c1239d0 con 0x7fa42c0b0770 2024-03-29T11:11:23.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.174+0000 7fa4237fe640 10 monclient(hunting): _negotiate 2024-03-29T11:11:23.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.174+0000 7fa4237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:23.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.174+0000 7fa4237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:23.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.174+0000 7fa4237fe640 10 monclient(hunting): my global_id is 4635 2024-03-29T11:11:23.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.174+0000 7fa4237fe640 1 -- 172.21.15.71:0/2336005193 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa42c145a50 con 0x7fa42c0b0770 2024-03-29T11:11:23.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.174+0000 7fa4237fe640 1 -- 172.21.15.71:0/2336005193 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3354060241 0 0) 0x7fa40c004eb0 con 0x7fa42c14a700 2024-03-29T11:11:23.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.174+0000 7fa4237fe640 1 -- 172.21.15.71:0/2336005193 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa42c1239d0 con 0x7fa42c14a700 2024-03-29T11:11:23.163 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.175+0000 7fa4237fe640 1 -- 172.21.15.71:0/2336005193 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fa4240018d0 con 0x7fa42c14a700 2024-03-29T11:11:23.163 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.175+0000 7fa4237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:23.163 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.175+0000 7fa4237fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:11:23.163 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.175+0000 7fa4237fe640 10 monclient(hunting): dump: 2024-03-29T11:11:23.163 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:11:23.163 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:23.163 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:23.163 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:23.163 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:23.163 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:23.163 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:11:23.163 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:11:23.163 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:11:23.163 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:23.163 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.175+0000 7fa4237fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:23.164 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.175+0000 7fa4237fe640 1 -- 172.21.15.71:0/2336005193 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1556651640 0 0) 0x7fa42c145a50 con 0x7fa42c0b0770 2024-03-29T11:11:23.164 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.175+0000 7fa4237fe640 1 -- 172.21.15.71:0/2336005193 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa40c004eb0 con 0x7fa42c0b0770 2024-03-29T11:11:23.164 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.175+0000 7fa4237fe640 1 -- 172.21.15.71:0/2336005193 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1772395377 0 0) 0x7fa42c1239d0 con 0x7fa42c14a700 2024-03-29T11:11:23.164 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.175+0000 7fa4237fe640 1 -- 172.21.15.71:0/2336005193 >> v1:172.21.15.137:6789/0 conn(0x7fa42c0ac5a0 legacy=0x7fa42c126690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:11:23.164 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.176+0000 7fa4237fe640 1 -- 172.21.15.71:0/2336005193 >> v1:172.21.15.71:6789/0 conn(0x7fa42c0b0770 legacy=0x7fa42c144e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:23.164 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.176+0000 7fa4237fe640 10 monclient: _finish_hunting 0 2024-03-29T11:11:23.165 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.176+0000 7fa4237fe640 1 monclient: found mon.c 2024-03-29T11:11:23.165 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.176+0000 7fa4237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:23.165 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.176+0000 7fa4237fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:11:23.165 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.176+0000 7fa4237fe640 1 -- 172.21.15.71:0/2336005193 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa42c145be0 con 0x7fa42c14a700 2024-03-29T11:11:23.165 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.177+0000 7fa4237fe640 10 monclient: _finish_auth 0 2024-03-29T11:11:23.165 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.177+0000 7fa4237fe640 10 monclient: _check_auth_tickets 2024-03-29T11:11:23.165 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.177+0000 7fa4237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:23.165 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.177+0000 7fa4310d0640 5 monclient: authenticate success, global_id 4703 2024-03-29T11:11:23.165 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.177+0000 7fa4310d0640 10 monclient: _renew_subs 2024-03-29T11:11:23.165 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.177+0000 7fa4310d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:11:23.165 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.177+0000 7fa4310d0640 1 -- 172.21.15.71:0/2336005193 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa42c126da0 con 0x7fa42c14a700 2024-03-29T11:11:23.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.177+0000 7fa4310d0640 10 client.4703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:23.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.177+0000 7fa4310d0640 10 monclient: _renew_subs 2024-03-29T11:11:23.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.177+0000 7fa4310d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:11:23.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.177+0000 7fa4310d0640 1 -- 172.21.15.71:0/2336005193 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa42c145730 con 0x7fa42c14a700 2024-03-29T11:11:23.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.177+0000 7fa4310d0640 1 librados: init done 2024-03-29T11:11:23.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.178+0000 7fa4237fe640 1 -- 172.21.15.71:0/2336005193 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa424001a30 con 0x7fa42c14a700 2024-03-29T11:11:23.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.178+0000 7fa4237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:23.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.178+0000 7fa4237fe640 1 -- 172.21.15.71:0/2336005193 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7fa424005110 con 0x7fa42c14a700 2024-03-29T11:11:23.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.178+0000 7fa4237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:23.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.178+0000 7fa4237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:23.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.178+0000 7fa4237fe640 10 monclient: dump: 2024-03-29T11:11:23.166 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:11:23.166 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:23.166 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:23.167 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:23.167 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:23.167 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:23.167 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:11:23.167 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:11:23.167 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:11:23.167 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:23.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.178+0000 7fa429d9b640 4 set_mon_vals no callback set 2024-03-29T11:11:23.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.178+0000 7fa4310d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:23.167 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.179+0000 7fa4310d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:23.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.179+0000 7fa4310d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:11:23.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.179+0000 7fa4310d0640 1 -- 172.21.15.71:0/2336005193 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa410005080 con 0x7fa42c14a700 2024-03-29T11:11:23.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.180+0000 7fa4237fe640 1 -- 172.21.15.71:0/2336005193 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7fa42401b0e0 con 0x7fa42c14a700 2024-03-29T11:11:23.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.184+0000 7fa4237fe640 1 -- 172.21.15.71:0/2336005193 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 45029510 0 0) 0x7fa424084e20 con 0x7fa42c14a700 2024-03-29T11:11:23.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.184+0000 7fa4237fe640 10 client.4703.objecter ms_dispatch 0x7fa42c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:11:23.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.185+0000 7fa4237fe640 3 client.4703.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:11:23.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.185+0000 7fa4237fe640 3 client.4703.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:11:23.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.185+0000 7fa4237fe640 20 client.4703.objecter dump_active .. 0 homeless 2024-03-29T11:11:23.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.186+0000 7fa42ad9d640 10 monclient: get_auth_request con 0x7fa40c06df80 auth_method 0 2024-03-29T11:11:23.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.187+0000 7fa4237fe640 10 client.4703.objecter ms_handle_connect 0x7fa40c06df80 2024-03-29T11:11:23.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.189+0000 7fa4237fe640 1 -- 172.21.15.71:0/2336005193 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa424050700 con 0x7fa42c14a700 2024-03-29T11:11:23.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.189+0000 7fa4237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:23.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.189+0000 7fa4237fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:23.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.506+0000 7fa4310d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:23.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.506+0000 7fa4310d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:23.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.506+0000 7fa4310d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:11:23.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.506+0000 7fa4310d0640 1 -- 172.21.15.71:0/2336005193 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa410005330 con 0x7fa42c14a700 2024-03-29T11:11:23.485 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:11:23.485 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:10:37.063243+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:10:37.063243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:23.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.507+0000 7fa4237fe640 1 -- 172.21.15.71:0/2336005193 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 577002532) 0x7fa424052630 con 0x7fa42c14a700 2024-03-29T11:11:23.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.507+0000 7fa4237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:23.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.507+0000 7fa4237fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:23.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.510+0000 7fa4310d0640 10 librados: watch_flush enter 2024-03-29T11:11:23.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.510+0000 7fa4310d0640 10 librados: watch_flush exit 2024-03-29T11:11:23.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.510+0000 7fa4310d0640 20 client.4703.objecter shutdown clearing up homeless session... 2024-03-29T11:11:23.489 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.510+0000 7fa4310d0640 10 client.4703.objecter successfully canceled tick 2024-03-29T11:11:23.489 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.511+0000 7fa4310d0640 1 -- 172.21.15.71:0/2336005193 >> v1:172.21.15.137:6816/973332665 conn(0x7fa40c06df80 legacy=0x7fa40c070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:23.489 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.511+0000 7fa4310d0640 10 monclient: shutdown 2024-03-29T11:11:23.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.511+0000 7fa4310d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:23.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.511+0000 7fa4310d0640 1 -- 172.21.15.71:0/2336005193 >> v1:172.21.15.71:6790/0 conn(0x7fa42c14a700 legacy=0x7fa42c1db700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:23.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.511+0000 7fa4310d0640 1 -- 172.21.15.71:0/2336005193 shutdown_connections 2024-03-29T11:11:23.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.511+0000 7fa4310d0640 1 -- 172.21.15.71:0/2336005193 >> 172.21.15.71:0/2336005193 conn(0x7fa42c06b660 msgr2=0x7fa42c06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:23.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.511+0000 7fa4310d0640 1 -- 172.21.15.71:0/2336005193 shutdown_connections 2024-03-29T11:11:23.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.511+0000 7fa4310d0640 1 -- 172.21.15.71:0/2336005193 wait complete. 2024-03-29T11:11:23.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:23.511+0000 7fa4310d0640 1 librados: shutdown 2024-03-29T11:11:23.504 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:11:30.317 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:30.462 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.483+0000 7f8e4dba3640 10 monclient: get_monmap_and_config 2024-03-29T11:11:30.463 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.483+0000 7f8e4dba3640 10 monclient: build_initial_monmap 2024-03-29T11:11:30.463 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.483+0000 7f8e4dba3640 10 monclient: monmap: 2024-03-29T11:11:30.463 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:11:30.463 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:30.463 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:11:30.484723+0000 2024-03-29T11:11:30.463 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:11:30.484723+0000 2024-03-29T11:11:30.463 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:30.463 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:30.463 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:11:30.463 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-c 2024-03-29T11:11:30.463 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:11:30.463 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:30.463 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.485+0000 7f8e4dba3640 1 Processor -- start 2024-03-29T11:11:30.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.485+0000 7f8e4dba3640 1 -- start start 2024-03-29T11:11:30.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.485+0000 7f8e4dba3640 10 monclient: init 2024-03-29T11:11:30.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.486+0000 7f8e4dba3640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:30.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.486+0000 7f8e4dba3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:11:30.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.486+0000 7f8e4dba3640 10 monclient(hunting): picked mon.noname-c con 0x7f8e40006410 addr v1:172.21.15.137:6789/0 2024-03-29T11:11:30.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.486+0000 7f8e4dba3640 10 monclient(hunting): picked mon.noname-b con 0x7f8e4002a640 addr v1:172.21.15.71:6790/0 2024-03-29T11:11:30.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.486+0000 7f8e4dba3640 10 monclient(hunting): picked mon.noname-a con 0x7f8e400327d0 addr v1:172.21.15.71:6789/0 2024-03-29T11:11:30.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.486+0000 7f8e4dba3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4002c820 con 0x7f8e400327d0 2024-03-29T11:11:30.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.486+0000 7f8e4dba3640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4002a120 con 0x7f8e40006410 2024-03-29T11:11:30.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.486+0000 7f8e4dba3640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e400c75b0 con 0x7f8e4002a640 2024-03-29T11:11:30.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.486+0000 7f8e4dba3640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:30.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.486+0000 7f8e4dba3640 10 monclient(hunting): authenticate will time out at 1446.744368s 2024-03-29T11:11:30.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.486+0000 7f8e47fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f8e4002a640 0x7f8e400320c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:52708/0 (socket says 172.21.15.71:52708) 2024-03-29T11:11:30.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.486+0000 7f8e47fff640 1 -- 172.21.15.71:0/3830999112 learned_addr learned my addr 172.21.15.71:0/3830999112 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:11:30.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.489+0000 7f8e477fe640 1 -- 172.21.15.71:0/3830999112 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982522428 0 0) 0x7f8e400c75b0 con 0x7f8e4002a640 2024-03-29T11:11:30.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.489+0000 7f8e477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:11:30.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.489+0000 7f8e477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:30.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.489+0000 7f8e477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:30.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.489+0000 7f8e477fe640 10 monclient(hunting): my global_id is 4706 2024-03-29T11:11:30.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.489+0000 7f8e477fe640 1 -- 172.21.15.71:0/3830999112 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e38003520 con 0x7f8e4002a640 2024-03-29T11:11:30.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.489+0000 7f8e477fe640 1 -- 172.21.15.71:0/3830999112 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1966444058 0 0) 0x7f8e38003520 con 0x7f8e4002a640 2024-03-29T11:11:30.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.489+0000 7f8e477fe640 1 -- 172.21.15.71:0/3830999112 >> v1:172.21.15.137:6789/0 conn(0x7f8e40006410 legacy=0x7f8e400c6ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:11:30.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.490+0000 7f8e477fe640 1 -- 172.21.15.71:0/3830999112 >> v1:172.21.15.71:6789/0 conn(0x7f8e400327d0 legacy=0x7f8e400c8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:30.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.490+0000 7f8e477fe640 10 monclient: _finish_hunting 0 2024-03-29T11:11:30.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.490+0000 7f8e477fe640 1 monclient: found mon.noname-b 2024-03-29T11:11:30.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.490+0000 7f8e477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:30.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.490+0000 7f8e477fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:11:30.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.490+0000 7f8e477fe640 1 -- 172.21.15.71:0/3830999112 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e40033110 con 0x7f8e4002a640 2024-03-29T11:11:30.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.490+0000 7f8e477fe640 10 monclient: _finish_auth 0 2024-03-29T11:11:30.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.490+0000 7f8e477fe640 10 monclient: _check_auth_tickets 2024-03-29T11:11:30.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.491+0000 7f8e477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:30.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.491+0000 7f8e4dba3640 5 monclient: authenticate success, global_id 4706 2024-03-29T11:11:30.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.491+0000 7f8e4dba3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:30.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.491+0000 7f8e477fe640 1 -- 172.21.15.71:0/3830999112 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f8e34001690 con 0x7f8e4002a640 2024-03-29T11:11:30.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.491+0000 7f8e477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:30.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.491+0000 7f8e477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:11:30.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.491+0000 7f8e477fe640 10 monclient: dump: 2024-03-29T11:11:30.472 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:11:30.472 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:30.473 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:30.473 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:30.473 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:30.473 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:30.473 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:11:30.473 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:11:30.473 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:11:30.473 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:30.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.491+0000 7f8e4dba3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:30.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.493+0000 7f8e477fe640 1 -- 172.21.15.71:0/3830999112 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e34001860 con 0x7f8e4002a640 2024-03-29T11:11:30.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.493+0000 7f8e477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:30.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.494+0000 7f8e4dba3640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:30.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.494+0000 7f8e4dba3640 4 set_mon_vals no callback set 2024-03-29T11:11:30.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.494+0000 7f8e477fe640 1 -- 172.21.15.71:0/3830999112 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f8e34001c70 con 0x7f8e4002a640 2024-03-29T11:11:30.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.494+0000 7f8e477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:30.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.494+0000 7f8e477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:30.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.494+0000 7f8e477fe640 10 monclient: dump: 2024-03-29T11:11:30.475 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:11:30.475 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:30.475 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:30.475 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:30.475 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:30.475 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:30.476 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:11:30.476 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:11:30.476 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:11:30.476 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:30.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.494+0000 7f8e4dba3640 10 monclient: shutdown 2024-03-29T11:11:30.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.494+0000 7f8e4dba3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:30.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.494+0000 7f8e4dba3640 1 -- 172.21.15.71:0/3830999112 >> v1:172.21.15.71:6790/0 conn(0x7f8e4002a640 legacy=0x7f8e400320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:30.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.495+0000 7f8e4dba3640 1 -- 172.21.15.71:0/3830999112 shutdown_connections 2024-03-29T11:11:30.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.495+0000 7f8e4dba3640 1 -- 172.21.15.71:0/3830999112 >> 172.21.15.71:0/3830999112 conn(0x7f8e400058f0 msgr2=0x7f8e40005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:30.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.495+0000 7f8e4dba3640 1 -- 172.21.15.71:0/3830999112 shutdown_connections 2024-03-29T11:11:30.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.495+0000 7f8e4dba3640 1 -- 172.21.15.71:0/3830999112 wait complete. 2024-03-29T11:11:30.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.497+0000 7f8e4dba3640 10 monclient: build_initial_monmap 2024-03-29T11:11:30.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.497+0000 7f8e4dba3640 10 monclient: monmap: 2024-03-29T11:11:30.478 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:11:30.478 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:30.478 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:11:30.478 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:11:30.478 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:30.478 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:30.478 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:11:30.478 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.noname-b 2024-03-29T11:11:30.478 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:11:30.478 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:30.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.498+0000 7f8e4dba3640 1 librados: starting msgr at 2024-03-29T11:11:30.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.498+0000 7f8e4dba3640 1 librados: starting objecter 2024-03-29T11:11:30.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.498+0000 7f8e4dba3640 1 Processor -- start 2024-03-29T11:11:30.479 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.499+0000 7f8e4dba3640 1 -- start start 2024-03-29T11:11:30.479 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.499+0000 7f8e4dba3640 1 librados: setting wanted keys 2024-03-29T11:11:30.479 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.499+0000 7f8e4dba3640 1 librados: calling monclient init 2024-03-29T11:11:30.479 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.499+0000 7f8e4dba3640 10 monclient: init 2024-03-29T11:11:30.479 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.499+0000 7f8e4dba3640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:30.479 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.500+0000 7f8e4dba3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:11:30.479 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.500+0000 7f8e4dba3640 10 monclient(hunting): picked mon.noname-b con 0x7f8e40006410 addr v1:172.21.15.137:6789/0 2024-03-29T11:11:30.479 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.500+0000 7f8e4dba3640 10 monclient(hunting): picked mon.noname-c con 0x7f8e4002a640 addr v1:172.21.15.71:6790/0 2024-03-29T11:11:30.480 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.501+0000 7f8e4dba3640 10 monclient(hunting): picked mon.noname-a con 0x7f8e400327d0 addr v1:172.21.15.71:6789/0 2024-03-29T11:11:30.480 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.501+0000 7f8e4dba3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e40031180 con 0x7f8e400327d0 2024-03-29T11:11:30.480 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.501+0000 7f8e4dba3640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e400d58a0 con 0x7f8e40006410 2024-03-29T11:11:30.480 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.501+0000 7f8e4dba3640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e400313d0 con 0x7f8e4002a640 2024-03-29T11:11:30.480 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.501+0000 7f8e4dba3640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:30.480 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.501+0000 7f8e4dba3640 10 monclient(hunting): authenticate will time out at 1446.759253s 2024-03-29T11:11:30.480 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.501+0000 7f8e47fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f8e4002a640 0x7f8e400bca60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:53592/0 (socket says 172.21.15.71:53592) 2024-03-29T11:11:30.480 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.501+0000 7f8e47fff640 1 -- 172.21.15.71:0/3463386120 learned_addr learned my addr 172.21.15.71:0/3463386120 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:11:30.480 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.501+0000 7f8e457fa640 10 client.?.objecter ms_handle_connect 0x7f8e4002a640 2024-03-29T11:11:30.480 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.501+0000 7f8e457fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:30.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.502+0000 7f8e457fa640 1 -- 172.21.15.71:0/3463386120 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 796296072 0 0) 0x7f8e400313d0 con 0x7f8e4002a640 2024-03-29T11:11:30.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.502+0000 7f8e457fa640 10 monclient(hunting): _negotiate 2024-03-29T11:11:30.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.502+0000 7f8e457fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:30.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.502+0000 7f8e457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:30.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.502+0000 7f8e457fa640 10 monclient(hunting): my global_id is 4709 2024-03-29T11:11:30.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.502+0000 7f8e457fa640 1 -- 172.21.15.71:0/3463386120 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e2c004eb0 con 0x7f8e4002a640 2024-03-29T11:11:30.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.503+0000 7f8e457fa640 1 -- 172.21.15.71:0/3463386120 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 858711435 0 0) 0x7f8e2c004eb0 con 0x7f8e4002a640 2024-03-29T11:11:30.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.503+0000 7f8e457fa640 1 -- 172.21.15.71:0/3463386120 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e400313d0 con 0x7f8e4002a640 2024-03-29T11:11:30.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.503+0000 7f8e457fa640 1 -- 172.21.15.71:0/3463386120 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f8e34001b70 con 0x7f8e4002a640 2024-03-29T11:11:30.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.503+0000 7f8e457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:30.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.503+0000 7f8e457fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:11:30.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.503+0000 7f8e457fa640 10 monclient(hunting): dump: 2024-03-29T11:11:30.482 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:11:30.482 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:30.482 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:30.482 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:30.482 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:30.482 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:30.482 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:11:30.482 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:11:30.482 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:11:30.482 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:30.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.503+0000 7f8e457fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:30.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.504+0000 7f8e457fa640 10 client.?.objecter ms_handle_connect 0x7f8e40006410 2024-03-29T11:11:30.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.504+0000 7f8e457fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:30.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.505+0000 7f8e457fa640 10 client.?.objecter ms_handle_connect 0x7f8e400327d0 2024-03-29T11:11:30.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.505+0000 7f8e457fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:30.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.505+0000 7f8e457fa640 1 -- 172.21.15.71:0/3463386120 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2267128576 0 0) 0x7f8e400d58a0 con 0x7f8e40006410 2024-03-29T11:11:30.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.505+0000 7f8e457fa640 10 monclient(hunting): _negotiate 2024-03-29T11:11:30.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.505+0000 7f8e457fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:30.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.505+0000 7f8e457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:30.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.505+0000 7f8e457fa640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:11:30.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.505+0000 7f8e457fa640 1 -- 172.21.15.71:0/3463386120 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e2c004eb0 con 0x7f8e40006410 2024-03-29T11:11:30.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.505+0000 7f8e457fa640 1 -- 172.21.15.71:0/3463386120 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17631484 0 0) 0x7f8e40031180 con 0x7f8e400327d0 2024-03-29T11:11:30.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.505+0000 7f8e457fa640 10 monclient(hunting): _negotiate 2024-03-29T11:11:30.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.505+0000 7f8e457fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:30.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.505+0000 7f8e457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:30.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.505+0000 7f8e457fa640 10 monclient(hunting): my global_id is 4641 2024-03-29T11:11:30.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.505+0000 7f8e457fa640 1 -- 172.21.15.71:0/3463386120 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e400d58a0 con 0x7f8e400327d0 2024-03-29T11:11:30.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.506+0000 7f8e457fa640 1 -- 172.21.15.71:0/3463386120 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1529863630 0 0) 0x7f8e400313d0 con 0x7f8e4002a640 2024-03-29T11:11:30.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.506+0000 7f8e457fa640 1 -- 172.21.15.71:0/3463386120 >> v1:172.21.15.137:6789/0 conn(0x7f8e40006410 legacy=0x7f8e400bc350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:30.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.506+0000 7f8e457fa640 1 -- 172.21.15.71:0/3463386120 >> v1:172.21.15.71:6789/0 conn(0x7f8e400327d0 legacy=0x7f8e40167710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:30.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.506+0000 7f8e457fa640 10 monclient: _finish_hunting 0 2024-03-29T11:11:30.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.506+0000 7f8e457fa640 1 monclient: found mon.c 2024-03-29T11:11:30.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.506+0000 7f8e457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:30.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.506+0000 7f8e457fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:11:30.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.506+0000 7f8e457fa640 1 -- 172.21.15.71:0/3463386120 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e40031a20 con 0x7f8e4002a640 2024-03-29T11:11:30.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.506+0000 7f8e457fa640 10 monclient: _finish_auth 0 2024-03-29T11:11:30.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.506+0000 7f8e457fa640 10 monclient: _check_auth_tickets 2024-03-29T11:11:30.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.506+0000 7f8e457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:30.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.506+0000 7f8e4dba3640 5 monclient: authenticate success, global_id 4709 2024-03-29T11:11:30.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.506+0000 7f8e4dba3640 10 monclient: _renew_subs 2024-03-29T11:11:30.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.506+0000 7f8e4dba3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:11:30.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.506+0000 7f8e4dba3640 1 -- 172.21.15.71:0/3463386120 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e400bd790 con 0x7f8e4002a640 2024-03-29T11:11:30.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.506+0000 7f8e4dba3640 10 client.4709.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:30.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.506+0000 7f8e4dba3640 10 monclient: _renew_subs 2024-03-29T11:11:30.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.506+0000 7f8e4dba3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:11:30.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.506+0000 7f8e4dba3640 1 -- 172.21.15.71:0/3463386120 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e400d61a0 con 0x7f8e4002a640 2024-03-29T11:11:30.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.506+0000 7f8e4dba3640 1 librados: init done 2024-03-29T11:11:30.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.507+0000 7f8e457fa640 1 -- 172.21.15.71:0/3463386120 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e34004bf0 con 0x7f8e4002a640 2024-03-29T11:11:30.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.507+0000 7f8e457fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:30.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.507+0000 7f8e457fa640 1 -- 172.21.15.71:0/3463386120 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 507903796 0 0) 0x7f8e34005040 con 0x7f8e4002a640 2024-03-29T11:11:30.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.507+0000 7f8e457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:30.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.507+0000 7f8e457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:30.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.507+0000 7f8e457fa640 10 monclient: dump: 2024-03-29T11:11:30.487 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:11:30.487 INFO:teuthology.misc.health.smithi071.stderr:fsid 8c8dcbb1-3a20-411a-b928-492e558bbb2b 2024-03-29T11:11:30.487 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:30.487 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T10:59:54.024963+0000 2024-03-29T11:11:30.487 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:30.487 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:11:30.487 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:11:30.487 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.137:6789/0 mon.b 2024-03-29T11:11:30.487 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:11:30.487 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:11:30.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.508+0000 7f8e4dba3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:30.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.508+0000 7f8e4dba3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:30.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.508+0000 7f8e4dba3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:11:30.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.508+0000 7f8e4dba3640 1 -- 172.21.15.71:0/3463386120 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e40031880 con 0x7f8e4002a640 2024-03-29T11:11:30.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.509+0000 7f8e467fc640 4 set_mon_vals no callback set 2024-03-29T11:11:30.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.510+0000 7f8e457fa640 1 -- 172.21.15.71:0/3463386120 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1188679321 0 0) 0x7f8e340051a0 con 0x7f8e4002a640 2024-03-29T11:11:30.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.513+0000 7f8e457fa640 1 -- 172.21.15.71:0/3463386120 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 45029510 0 0) 0x7f8e34084d30 con 0x7f8e4002a640 2024-03-29T11:11:30.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.513+0000 7f8e457fa640 10 client.4709.objecter ms_dispatch 0x7f8e48000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:11:30.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.513+0000 7f8e457fa640 3 client.4709.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:11:30.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.513+0000 7f8e457fa640 3 client.4709.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:11:30.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.513+0000 7f8e457fa640 20 client.4709.objecter dump_active .. 0 homeless 2024-03-29T11:11:30.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.513+0000 7f8e4cba1640 10 monclient: get_auth_request con 0x7f8e2c06dd80 auth_method 0 2024-03-29T11:11:30.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.514+0000 7f8e457fa640 10 client.4709.objecter ms_handle_connect 0x7f8e2c06dd80 2024-03-29T11:11:30.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.519+0000 7f8e457fa640 1 -- 172.21.15.71:0/3463386120 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8e40031880 con 0x7f8e4002a640 2024-03-29T11:11:30.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.519+0000 7f8e457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:30.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.519+0000 7f8e457fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:30.818 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.840+0000 7f8e4dba3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:30.818 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.840+0000 7f8e4dba3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:30.818 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.840+0000 7f8e4dba3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:11:30.818 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.840+0000 7f8e4dba3640 1 -- 172.21.15.71:0/3463386120 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e40031880 con 0x7f8e4002a640 2024-03-29T11:11:30.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.841+0000 7f8e457fa640 1 -- 172.21.15.71:0/3463386120 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 577002532) 0x7f8e40031880 con 0x7f8e4002a640 2024-03-29T11:11:30.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.841+0000 7f8e457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:30.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.841+0000 7f8e457fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:30.819 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:11:30.819 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"8c8dcbb1-3a20-411a-b928-492e558bbb2b","created":"2024-03-29T11:00:32.140252+0000","modified":"2024-03-29T11:10:37.063243+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:10:37.063243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6ae3879f-03d5-4971-92a9-8a7dce1e877d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5c6efb84-6592-41d7-b4f1-b4a33d2d2f83","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ef7c1ae2-e54a-42ce-8ab5-9d08c2a82d05","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"395ce08b-6e75-4493-ad8b-3ca9763c444e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9434f80-2148-42b6-a4d9-ca9b055b61c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"21446639-1b33-4ae5-9997-6fdfad0dc87e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aa3fdfbe-4e06-411f-9b6e-88732ca48f90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a7c11f95-50b1-4b51-aba5-4fe2e78c0f24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:30.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.844+0000 7f8e4dba3640 10 librados: watch_flush enter 2024-03-29T11:11:30.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.844+0000 7f8e4dba3640 10 librados: watch_flush exit 2024-03-29T11:11:30.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.844+0000 7f8e4dba3640 20 client.4709.objecter shutdown clearing up homeless session... 2024-03-29T11:11:30.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.844+0000 7f8e4dba3640 10 client.4709.objecter successfully canceled tick 2024-03-29T11:11:30.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.844+0000 7f8e4dba3640 1 -- 172.21.15.71:0/3463386120 >> v1:172.21.15.137:6816/973332665 conn(0x7f8e2c06dd80 legacy=0x7f8e2c070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:30.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.844+0000 7f8e4dba3640 10 monclient: shutdown 2024-03-29T11:11:30.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.844+0000 7f8e4dba3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:30.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.844+0000 7f8e4dba3640 1 -- 172.21.15.71:0/3463386120 >> v1:172.21.15.71:6790/0 conn(0x7f8e4002a640 legacy=0x7f8e400bca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:30.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.844+0000 7f8e4dba3640 1 -- 172.21.15.71:0/3463386120 shutdown_connections 2024-03-29T11:11:30.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.845+0000 7f8e4dba3640 1 -- 172.21.15.71:0/3463386120 >> 172.21.15.71:0/3463386120 conn(0x7f8e400058f0 msgr2=0x7f8e400043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:30.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.845+0000 7f8e4dba3640 1 -- 172.21.15.71:0/3463386120 shutdown_connections 2024-03-29T11:11:30.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.845+0000 7f8e4dba3640 1 -- 172.21.15.71:0/3463386120 wait complete. 2024-03-29T11:11:30.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:11:30.845+0000 7f8e4dba3640 1 librados: shutdown 2024-03-29T11:11:30.839 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:11:30.839 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:11:30.841 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:11:30.841 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:11:30.842 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:11:30.874 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:11:30.874 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:11:30.874 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:11:30.953 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:11:30.953 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:11:30.953 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:11:31.023 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:11:31.023 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:11:31.023 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:11:31.097 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:11:31.097 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:11:31.097 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:11:31.140 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:11:31.141 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:11:31.141 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:11:31.172 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:11:31.172 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:11:31.173 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:11:31.239 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:11:31.239 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:11:31.239 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:11:31.311 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:11:31.311 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:11:31.311 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:11:31.311 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:11:31.335 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:11:31.335 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:11:31.335 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:11:31.377 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:11:31.377 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:11:31.377 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:11:31.377 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:11:31.389 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:11:31.389 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:11:31.389 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:11:31.441 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:11:31.442 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:11:31.442 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:11:31.453 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:11:31.453 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:11:31.453 DEBUG:teuthology.orchestra.run.smithi071:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:11:31.629 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi071.front.sepia.ceph.com 2024-03-29T11:11:31.629 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:11:33.941 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi071.front.sepia.ceph.com 2024-03-29T11:11:33.942 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:11:34.062 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi071.front.sepia.ceph.com 2024-03-29T11:11:34.062 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:11:34.119 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi071.front.sepia.ceph.com 2024-03-29T11:11:34.120 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:11:34.175 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi137.front.sepia.ceph.com 2024-03-29T11:11:34.175 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:11:34.569 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi137.front.sepia.ceph.com 2024-03-29T11:11:34.569 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:11:34.671 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi137.front.sepia.ceph.com 2024-03-29T11:11:34.672 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:11:34.737 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi137.front.sepia.ceph.com 2024-03-29T11:11:34.738 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:11:34.803 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:11:34.805 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622670/data/mon.a.tgz 2024-03-29T11:11:34.805 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2024-03-29T11:11:34.823 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.NHnbqXhk0D 2024-03-29T11:11:34.824 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.NHnbqXhk0D 2024-03-29T11:11:34.940 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.NHnbqXhk0D 2024-03-29T11:11:35.028 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.NHnbqXhk0D is 211KB 2024-03-29T11:11:35.123 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.NHnbqXhk0D 2024-03-29T11:11:35.138 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622670/data/mon.c.tgz 2024-03-29T11:11:35.138 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2024-03-29T11:11:35.194 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.YG5yshRSwK 2024-03-29T11:11:35.195 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.YG5yshRSwK 2024-03-29T11:11:35.295 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.YG5yshRSwK 2024-03-29T11:11:35.379 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.YG5yshRSwK is 211KB 2024-03-29T11:11:35.479 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.YG5yshRSwK 2024-03-29T11:11:35.493 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622670/data/mon.b.tgz 2024-03-29T11:11:35.493 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-03-29T11:11:35.512 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.YUWWj0m23Q 2024-03-29T11:11:35.513 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.YUWWj0m23Q 2024-03-29T11:11:35.632 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.YUWWj0m23Q 2024-03-29T11:11:35.720 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.YUWWj0m23Q is 211KB 2024-03-29T11:11:35.816 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.YUWWj0m23Q 2024-03-29T11:11:35.831 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:11:35.832 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:11:35.834 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:11:35.914 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:11:35.952 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:11:35.953 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:11:35.955 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622670/remote/smithi071/crash 2024-03-29T11:11:35.956 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:11:35.983 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622670/remote/smithi137/crash 2024-03-29T11:11:35.984 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:11:36.019 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:11:36.019 DEBUG:teuthology.orchestra.run.smithi071:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:11:36.028 DEBUG:teuthology.orchestra.run.smithi137:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:11:36.102 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34277.log 2024-03-29T11:11:36.102 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:11:36.102 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34357.log 2024-03-29T11:11:36.102 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34277.log: /var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34477.log 2024-03-29T11:11:36.102 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34557.log 2024-03-29T11:11:36.102 INFO:teuthology.orchestra.run.smithi137.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34277.log.gz 2024-03-29T11:11:36.103 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34477.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:11:36.103 INFO:teuthology.orchestra.run.smithi137.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34357.log.gz 2024-03-29T11:11:36.103 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34437.log 2024-03-29T11:11:36.103 INFO:teuthology.orchestra.run.smithi137.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34477.log.gz 2024-03-29T11:11:36.103 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:11:36.103 INFO:teuthology.orchestra.run.smithi137.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34557.log.gz 2024-03-29T11:11:36.103 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34437.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:11:36.103 INFO:teuthology.orchestra.run.smithi137.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:11:36.103 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:11:36.103 INFO:teuthology.orchestra.run.smithi137.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34437.log.gz 2024-03-29T11:11:36.103 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34517.log 2024-03-29T11:11:36.103 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34317.log 2024-03-29T11:11:36.103 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34517.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:11:36.103 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34397.log 2024-03-29T11:11:36.104 INFO:teuthology.orchestra.run.smithi137.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34517.log.gz 2024-03-29T11:11:36.104 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:11:36.104 INFO:teuthology.orchestra.run.smithi137.stderr: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.34317.log.gz -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:11:36.104 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:11:36.104 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34397.log: /var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34397.log.gz 2024-03-29T11:11:36.104 INFO:teuthology.orchestra.run.smithi137.stderr: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:11:36.104 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40740.log 2024-03-29T11:11:36.105 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40230.log 2024-03-29T11:11:36.105 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40740.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:11:36.105 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39060.log 2024-03-29T11:11:36.105 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-mon.a.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40740.log.gz 2024-03-29T11:11:36.105 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38224.log 2024-03-29T11:11:36.105 INFO:teuthology.orchestra.run.smithi071.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40230.log.gz 2024-03-29T11:11:36.105 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41013.log 2024-03-29T11:11:36.105 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38473.log 2024-03-29T11:11:36.105 INFO:teuthology.orchestra.run.smithi071.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39060.log.gz 2024-03-29T11:11:36.105 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39840.log 2024-03-29T11:11:36.105 INFO:teuthology.orchestra.run.smithi071.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38224.log.gz 2024-03-29T11:11:36.105 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39294.log 2024-03-29T11:11:36.106 INFO:teuthology.orchestra.run.smithi071.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41013.log.gz 2024-03-29T11:11:36.106 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39177.log 2024-03-29T11:11:36.106 INFO:teuthology.orchestra.run.smithi071.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38473.log.gz 2024-03-29T11:11:36.106 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39294.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39840.log.gz 2024-03-29T11:11:36.106 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41329.log 2024-03-29T11:11:36.106 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41052.log 2024-03-29T11:11:36.106 INFO:teuthology.orchestra.run.smithi071.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39294.log.gz 2024-03-29T11:11:36.106 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38982.log 2024-03-29T11:11:36.106 INFO:teuthology.orchestra.run.smithi071.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39177.log.gz 2024-03-29T11:11:36.106 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39645.log 2024-03-29T11:11:36.106 INFO:teuthology.orchestra.run.smithi071.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41329.log.gz 2024-03-29T11:11:36.106 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38982.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41052.log.gz 2024-03-29T11:11:36.106 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40191.log 2024-03-29T11:11:36.106 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40542.log 2024-03-29T11:11:36.107 INFO:teuthology.orchestra.run.smithi071.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38982.log.gz 2024-03-29T11:11:36.107 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38943.log 2024-03-29T11:11:36.107 INFO:teuthology.orchestra.run.smithi071.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39645.log.gz 2024-03-29T11:11:36.107 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39762.log 2024-03-29T11:11:36.107 INFO:teuthology.orchestra.run.smithi071.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40191.log.gz 2024-03-29T11:11:36.107 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40935.log 2024-03-29T11:11:36.107 INFO:teuthology.orchestra.run.smithi071.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40542.log.gz 2024-03-29T11:11:36.107 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41249.log 2024-03-29T11:11:36.107 INFO:teuthology.orchestra.run.smithi071.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38943.log.gz 2024-03-29T11:11:36.107 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41602.log 2024-03-29T11:11:36.107 INFO:teuthology.orchestra.run.smithi071.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39762.log.gz 2024-03-29T11:11:36.107 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41249.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40935.log.gz 2024-03-29T11:11:36.107 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41169.log 2024-03-29T11:11:36.107 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39255.log 2024-03-29T11:11:36.108 INFO:teuthology.orchestra.run.smithi071.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41249.log.gz 2024-03-29T11:11:36.108 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38748.log 2024-03-29T11:11:36.108 INFO:teuthology.orchestra.run.smithi071.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41602.log.gz 2024-03-29T11:11:36.108 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39255.log: gzip 85.6% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.41169.log.gz -- 2024-03-29T11:11:36.108 INFO:teuthology.orchestra.run.smithi071.stderr: /var/log/ceph/ceph-client.admin.40662.log 2024-03-29T11:11:36.108 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34998.log 2024-03-29T11:11:36.108 INFO:teuthology.orchestra.run.smithi071.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39255.log.gz 2024-03-29T11:11:36.108 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41641.log 2024-03-29T11:11:36.108 INFO:teuthology.orchestra.run.smithi071.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38748.log.gz 2024-03-29T11:11:36.108 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.34998.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:11:36.108 INFO:teuthology.orchestra.run.smithi071.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40662.log.gz 2024-03-29T11:11:36.108 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40152.log 2024-03-29T11:11:36.108 INFO:teuthology.orchestra.run.smithi071.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34998.log.gz 2024-03-29T11:11:36.108 INFO:teuthology.orchestra.run.smithi071.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41641.log.gz 2024-03-29T11:11:36.109 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38631.log 2024-03-29T11:11:36.109 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.40152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39099.log 2024-03-29T11:11:36.109 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38631.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40152.log.gz 2024-03-29T11:11:36.109 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38434.log 2024-03-29T11:11:36.109 INFO:teuthology.orchestra.run.smithi071.stderr: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:11:36.109 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41524.log 2024-03-29T11:11:36.109 INFO:teuthology.orchestra.run.smithi071.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38631.log.gz 2024-03-29T11:11:36.109 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38904.log 2024-03-29T11:11:36.109 INFO:teuthology.orchestra.run.smithi071.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39099.log.gz 2024-03-29T11:11:36.109 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39489.log 2024-03-29T11:11:36.109 INFO:teuthology.orchestra.run.smithi071.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38434.log.gz 2024-03-29T11:11:36.109 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39567.log 2024-03-29T11:11:36.109 INFO:teuthology.orchestra.run.smithi071.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41524.log.gz 2024-03-29T11:11:36.109 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38553.log 2024-03-29T11:11:36.109 INFO:teuthology.orchestra.run.smithi071.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38904.log.gz 2024-03-29T11:11:36.110 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41368.log 2024-03-29T11:11:36.110 INFO:teuthology.orchestra.run.smithi071.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39489.log.gz 2024-03-29T11:11:36.110 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38553.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.39567.log.gz 2024-03-29T11:11:36.110 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39333.log 2024-03-29T11:11:36.110 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41446.log 2024-03-29T11:11:36.110 INFO:teuthology.orchestra.run.smithi071.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38553.log.gz 2024-03-29T11:11:36.110 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39333.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:11:36.110 INFO:teuthology.orchestra.run.smithi071.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.41368.log.gz 2024-03-29T11:11:36.110 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39528.log 2024-03-29T11:11:36.110 INFO:teuthology.orchestra.run.smithi071.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39333.log.gz 2024-03-29T11:11:36.110 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39879.log 2024-03-29T11:11:36.110 INFO:teuthology.orchestra.run.smithi071.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41446.log.gz 2024-03-29T11:11:36.110 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38592.log 2024-03-29T11:11:36.110 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40074.log 2024-03-29T11:11:36.111 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39879.log: /var/log/ceph/ceph-client.admin.38592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39138.log 2024-03-29T11:11:36.111 INFO:teuthology.orchestra.run.smithi071.stderr: 86.9% 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39879.log.gz 2024-03-29T11:11:36.111 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39528.log.gz 2024-03-29T11:11:36.111 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40074.log: gzip -5 --verbose -- 86.9% /var/log/ceph/ceph-client.admin.40425.log -- replaced with /var/log/ceph/ceph-client.admin.38592.log.gz 2024-03-29T11:11:36.111 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:11:36.111 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38787.log 2024-03-29T11:11:36.111 INFO:teuthology.orchestra.run.smithi071.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40074.log.gz 2024-03-29T11:11:36.111 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40425.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39138.log.gz 2024-03-29T11:11:36.111 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40701.log 2024-03-29T11:11:36.111 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41680.log 2024-03-29T11:11:36.111 INFO:teuthology.orchestra.run.smithi071.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40425.log.gz 2024-03-29T11:11:36.111 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40701.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38787.log.gz 2024-03-29T11:11:36.111 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34906.log 2024-03-29T11:11:36.111 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40896.log 2024-03-29T11:11:36.111 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41680.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40701.log.gz 2024-03-29T11:11:36.112 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.34906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38865.log 2024-03-29T11:11:36.112 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40896.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41680.log.gz 2024-03-29T11:11:36.112 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39957.log 2024-03-29T11:11:36.112 INFO:teuthology.orchestra.run.smithi071.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34906.log.gz 2024-03-29T11:11:36.112 INFO:teuthology.orchestra.run.smithi071.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40896.log.gz 2024-03-29T11:11:36.112 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41485.log 2024-03-29T11:11:36.112 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:11:36.112 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39957.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38865.log.gz/var/log/ceph/ceph-client.admin.41485.log: 2024-03-29T11:11:36.112 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40857.log 2024-03-29T11:11:36.112 INFO:teuthology.orchestra.run.smithi071.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39957.log.gz 2024-03-29T11:11:36.112 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39216.log 2024-03-29T11:11:36.112 INFO:teuthology.orchestra.run.smithi071.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41485.log.gz 2024-03-29T11:11:36.112 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40308.log 2024-03-29T11:11:36.112 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39216.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40857.log.gz 2024-03-29T11:11:36.113 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38302.log 2024-03-29T11:11:36.113 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38670.log 2024-03-29T11:11:36.113 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40308.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39216.log.gz 2024-03-29T11:11:36.113 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40503.log 2024-03-29T11:11:36.113 INFO:teuthology.orchestra.run.smithi071.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40308.log.gz 2024-03-29T11:11:36.113 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38670.log: 86.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.38302.log.gz 2024-03-29T11:11:36.113 INFO:teuthology.orchestra.run.smithi071.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39021.log 2024-03-29T11:11:36.113 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41563.log 2024-03-29T11:11:36.113 INFO:teuthology.orchestra.run.smithi071.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38670.log.gz 2024-03-29T11:11:36.113 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39801.log 2024-03-29T11:11:36.113 INFO:teuthology.orchestra.run.smithi071.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40503.log.gz 2024-03-29T11:11:36.113 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41563.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39021.log.gz 2024-03-29T11:11:36.113 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40818.log 2024-03-29T11:11:36.113 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40464.log 2024-03-29T11:11:36.113 INFO:teuthology.orchestra.run.smithi071.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41563.log.gz 2024-03-29T11:11:36.114 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38826.log 2024-03-29T11:11:36.114 INFO:teuthology.orchestra.run.smithi071.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39801.log.gz 2024-03-29T11:11:36.114 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40464.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40818.log.gz 2024-03-29T11:11:36.114 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40779.log 2024-03-29T11:11:36.114 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38263.log 2024-03-29T11:11:36.114 INFO:teuthology.orchestra.run.smithi071.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40464.log.gz 2024-03-29T11:11:36.114 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38826.log: /var/log/ceph/ceph-client.admin.40779.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:11:36.114 INFO:teuthology.orchestra.run.smithi071.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.38826.log.gz 2024-03-29T11:11:36.114 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38356.log 2024-03-29T11:11:36.114 INFO:teuthology.orchestra.run.smithi071.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40779.log.gz 2024-03-29T11:11:36.114 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39411.log 2024-03-29T11:11:36.114 INFO:teuthology.orchestra.run.smithi071.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38263.log.gz 2024-03-29T11:11:36.114 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38709.log 2024-03-29T11:11:36.114 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39411.log: gzip -5 --verbose -- 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38356.log.gz 2024-03-29T11:11:36.115 INFO:teuthology.orchestra.run.smithi071.stderr: /var/log/ceph/ceph-client.admin.39996.log 2024-03-29T11:11:36.115 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38709.log: 86.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40974.log 2024-03-29T11:11:36.115 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39411.log.gz 2024-03-29T11:11:36.115 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39996.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38709.log.gz 2024-03-29T11:11:36.115 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38514.log 2024-03-29T11:11:36.115 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40269.log 2024-03-29T11:11:36.115 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40974.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39996.log.gz 2024-03-29T11:11:36.115 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39450.log 2024-03-29T11:11:36.115 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41210.log 2024-03-29T11:11:36.115 INFO:teuthology.orchestra.run.smithi071.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38514.log.gz 2024-03-29T11:11:36.115 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39450.log: 86.7% 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40269.log.gz 2024-03-29T11:11:36.115 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40974.log.gz 2024-03-29T11:11:36.115 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:11:36.115 INFO:teuthology.orchestra.run.smithi071.stderr: 86.7%/var/log/ceph/ceph-client.admin.41210.log: -- replaced with /var/log/ceph/ceph-client.admin.39450.log.gz 2024-03-29T11:11:36.115 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40347.log 2024-03-29T11:11:36.116 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.0.log: 87.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40386.log 2024-03-29T11:11:36.116 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41210.log.gz 2024-03-29T11:11:36.116 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40347.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:11:36.116 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.40386.log: -- 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40347.log.gz 2024-03-29T11:11:36.116 INFO:teuthology.orchestra.run.smithi071.stderr: /var/log/ceph/ceph-osd.2.log 2024-03-29T11:11:36.116 INFO:teuthology.orchestra.run.smithi071.stderr:gzip 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40386.log.gz 2024-03-29T11:11:36.116 INFO:teuthology.orchestra.run.smithi071.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40620.log 2024-03-29T11:11:36.116 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34604.log 2024-03-29T11:11:36.116 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40620.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40620.log.gz 2024-03-29T11:11:36.116 INFO:teuthology.orchestra.run.smithi071.stderr: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:11:36.116 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40113.log 2024-03-29T11:11:36.116 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.34604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41290.log 2024-03-29T11:11:36.116 INFO:teuthology.orchestra.run.smithi071.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.34604.log.gz 2024-03-29T11:11:36.116 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41091.log 2024-03-29T11:11:36.117 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41290.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40113.log.gz 2024-03-29T11:11:36.117 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39723.log 2024-03-29T11:11:36.117 INFO:teuthology.orchestra.run.smithi071.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41290.log.gz 2024-03-29T11:11:36.117 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40581.log 2024-03-29T11:11:36.117 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39723.log: 93.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:11:36.117 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41130.log 2024-03-29T11:11:36.117 INFO:teuthology.orchestra.run.smithi071.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41091.log.gz 2024-03-29T11:11:36.117 INFO:teuthology.orchestra.run.smithi071.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39723.log.gz/var/log/ceph/ceph-client.admin.40581.log: 2024-03-29T11:11:36.117 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39918.log 2024-03-29T11:11:36.117 INFO:teuthology.orchestra.run.smithi071.stderr: 85.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39606.log 2024-03-29T11:11:36.117 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40581.log.gz 2024-03-29T11:11:36.117 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41130.log: /var/log/ceph/ceph-client.admin.39918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40035.log 2024-03-29T11:11:36.117 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39606.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41130.log.gz 2024-03-29T11:11:36.117 INFO:teuthology.orchestra.run.smithi071.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39918.log.gz 2024-03-29T11:11:36.117 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28772.log 2024-03-29T11:11:36.118 INFO:teuthology.orchestra.run.smithi071.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39606.log.gz 2024-03-29T11:11:36.118 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.40035.log: /var/log/ceph/ceph-client.admin.39684.log 2024-03-29T11:11:36.118 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph.tmp-client.admin.28772.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28772.log.gz 2024-03-29T11:11:36.118 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39372.log 2024-03-29T11:11:36.118 INFO:teuthology.orchestra.run.smithi071.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40035.log.gz 2024-03-29T11:11:36.118 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41407.log 2024-03-29T11:11:36.118 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39372.log: /var/log/ceph/ceph-client.admin.39684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38395.log 2024-03-29T11:11:36.118 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41407.log: 86.2% 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39684.log.gz 2024-03-29T11:11:36.118 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39372.log.gz 2024-03-29T11:11:36.118 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- 85.7% /var/log/ceph/ceph-mon.c.log 2024-03-29T11:11:36.118 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41407.log.gz 2024-03-29T11:11:36.118 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38395.log: /var/log/ceph/ceph-mon.c.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38395.log.gz 2024-03-29T11:11:36.163 INFO:teuthology.orchestra.run.smithi137.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:11:47.983 INFO:teuthology.orchestra.run.smithi137.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:11:48.276 INFO:teuthology.orchestra.run.smithi137.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:11:48.346 INFO:teuthology.orchestra.run.smithi137.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:11:52.536 INFO:teuthology.orchestra.run.smithi137.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:11:53.717 INFO:teuthology.orchestra.run.smithi071.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:11:54.132 INFO:teuthology.orchestra.run.smithi071.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:11:54.356 INFO:teuthology.orchestra.run.smithi071.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:11:54.494 INFO:teuthology.orchestra.run.smithi071.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:12:23.378 INFO:teuthology.orchestra.run.smithi071.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:12:29.292 INFO:teuthology.orchestra.run.smithi071.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:12:29.294 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:12:29.294 INFO:teuthology.orchestra.run.smithi071.stderr:real 0m53.255s 2024-03-29T11:12:29.295 INFO:teuthology.orchestra.run.smithi071.stderr:user 2m46.794s 2024-03-29T11:12:29.295 INFO:teuthology.orchestra.run.smithi071.stderr:sys 0m5.879s 2024-03-29T11:12:45.856 INFO:teuthology.orchestra.run.smithi137.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:12:45.857 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-03-29T11:12:45.857 INFO:teuthology.orchestra.run.smithi137.stderr:real 1m9.780s 2024-03-29T11:12:45.857 INFO:teuthology.orchestra.run.smithi137.stderr:user 1m57.960s 2024-03-29T11:12:45.857 INFO:teuthology.orchestra.run.smithi137.stderr:sys 0m4.362s 2024-03-29T11:12:45.858 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:12:45.858 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622670/remote/smithi071/log 2024-03-29T11:12:45.859 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:13:10.023 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622670/remote/smithi137/log 2024-03-29T11:13:10.024 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:13:30.943 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:13:31.167 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=cde3098c5bca4cb2b11c3306f689695a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:13:31.170 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:13:31.186 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:13:31.234 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:13:31.235 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:13:31.250 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:13:31.338 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:13:31.350 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:13:31.351 DEBUG:teuthology.orchestra.run.smithi071:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:13:31.355 DEBUG:teuthology.orchestra.run.smithi137:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:13:31.371 INFO:teuthology.orchestra.run.smithi071.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:13:31.373 INFO:teuthology.orchestra.run.smithi137.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:13:31.375 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:13:31.376 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2024-03-29T11:13:31.376 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 51 +408us[ +409us] +/- 74ms 2024-03-29T11:13:31.376 INFO:teuthology.orchestra.run.smithi071.stdout:^* hv02.front.sepia.ceph.com 3 6 377 50 -185us[ -185us] +/- 69ms 2024-03-29T11:13:31.376 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 50 -185us[ -185us] +/- 69ms 2024-03-29T11:13:31.376 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:13:31.378 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:13:31.378 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2024-03-29T11:13:31.378 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 51 +414us[ +414us] +/- 74ms 2024-03-29T11:13:31.378 INFO:teuthology.orchestra.run.smithi137.stdout:^* hv02.front.sepia.ceph.com 3 6 377 52 -190us[ -191us] +/- 69ms 2024-03-29T11:13:31.378 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 50 -184us[ -184us] +/- 69ms 2024-03-29T11:13:31.378 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:13:31.380 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:13:31.390 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:13:31.391 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:13:31.417 DEBUG:teuthology.orchestra.run.smithi071:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:13:31.488 DEBUG:teuthology.orchestra.run.smithi137:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:13:31.561 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:13:31.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:13:31.593 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:13:31.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:13:31.625 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:13:31.637 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:13:31.684 INFO:teuthology.task.internal:Duration was 1228.877706 seconds 2024-03-29T11:13:31.684 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:13:31.710 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:13:31.710 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:13:31.713 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:13:31.757 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:13:31.758 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:13:32.058 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:13:32.059 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi071.front.sepia.ceph.com 2024-03-29T11:13:32.059 DEBUG:teuthology.orchestra.run.smithi071:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:13:32.080 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi137.front.sepia.ceph.com 2024-03-29T11:13:32.080 DEBUG:teuthology.orchestra.run.smithi137:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:13:32.108 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:13:32.108 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:13:32.123 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:13:32.178 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:13:32.178 DEBUG:teuthology.orchestra.run.smithi071:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:13:32.193 DEBUG:teuthology.orchestra.run.smithi137:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:13:32.315 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:13:32.326 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:13:32.326 DEBUG:teuthology.orchestra.run.smithi071:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:13:32.354 DEBUG:teuthology.orchestra.run.smithi137:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:13:32.382 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:13:32.394 DEBUG:teuthology.orchestra.run.smithi071:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:13:32.397 DEBUG:teuthology.orchestra.run.smithi137:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:13:32.419 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = core 2024-03-29T11:13:32.448 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = core 2024-03-29T11:13:32.474 DEBUG:teuthology.orchestra.run.smithi071:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:13:32.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:13:32.501 DEBUG:teuthology.orchestra.run.smithi137:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:13:32.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:13:32.528 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:13:32.538 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:13:32.539 DEBUG:teuthology.misc:Transferring archived files from smithi071:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622670/remote/smithi071 2024-03-29T11:13:32.539 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:13:32.579 DEBUG:teuthology.misc:Transferring archived files from smithi137:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622670/remote/smithi137 2024-03-29T11:13:32.580 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:13:32.616 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:13:32.616 DEBUG:teuthology.orchestra.run.smithi071:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:13:32.618 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:13:32.666 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:13:32.680 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:13:32.680 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:13:32.702 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:13:32.702 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:13:32.704 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:13:32.717 INFO:teuthology.orchestra.run.smithi071.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:13 /home/ubuntu/cephtest 2024-03-29T11:13:32.720 INFO:teuthology.orchestra.run.smithi137.stdout: 265844 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:13 /home/ubuntu/cephtest 2024-03-29T11:13:32.722 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:13:32.737 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:13:32.800 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:13:32.828 DEBUG:teuthology.nuke:shortname: smithi071 2024-03-29T11:13:32.829 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:13:32.851 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622670', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:44:35.638200', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOcs4iTZhsRuZeUJNRGglRkljVWYqSRuS0hdE/+KYKineMcL/m/NgmcjEif3aIR8bvEZPniQYgZqCuqPgWkrFpw='} 2024-03-29T11:13:32.856 DEBUG:teuthology.nuke:shortname: smithi137 2024-03-29T11:13:32.856 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:13:32.878 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622670', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:44:35.640092', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLfH/EJ2PkJNYZoxy9WM5+7uIdEXJwM+CpijWnhtSm97Q9XamauYkrsr/bGxV6otwT2Zv1yDdYVkkOOT1KktJe0='} 2024-03-29T11:13:32.914 INFO:teuthology.orchestra.console.smithi071:Power off 2024-03-29T11:13:32.915 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:13:32.942 DEBUG:teuthology.orchestra.console.smithi071:power off output: Chassis Power Control: Down/Off 2024-03-29T11:13:32.943 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:13:32.960 INFO:teuthology.orchestra.console.smithi137:Power off 2024-03-29T11:13:32.960 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:13:32.975 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on 2024-03-29T11:13:32.989 DEBUG:teuthology.orchestra.console.smithi137:power off output: Chassis Power Control: Down/Off 2024-03-29T11:13:32.990 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:13:33.019 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is on 2024-03-29T11:13:36.976 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:13:37.020 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:13:37.088 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on 2024-03-29T11:13:37.130 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is on 2024-03-29T11:13:41.092 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:13:41.145 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:13:41.207 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on 2024-03-29T11:13:41.257 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is off 2024-03-29T11:13:41.358 INFO:teuthology.orchestra.console.smithi137:Power off completed 2024-03-29T11:13:45.210 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:13:45.323 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is off 2024-03-29T11:13:45.424 INFO:teuthology.orchestra.console.smithi071:Power off completed 2024-03-29T11:13:45.528 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} duration: 1228.8777055740356 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=cde3098c5bca4cb2b11c3306f689695a status: fail success: false 2024-03-29T11:13:45.529 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:13:45.609 INFO:teuthology.run:FAIL